From Logits to the Next Token

In From Raw Text to Token IDs and Embeddings, you traced a prompt through tokenization and embedding lookup to arrive at the input tensor [1, 3, 4] — one batch, three tokens, four features per token. That tensor enters the transformer’s layers. When the model finishes its forward pass, it produces one output vector per position. The last position’s vector is projected to a score for every token in the vocabulary. Those scores are called logits.

This lesson explains what logits are, how they become probabilities, and how different decoding strategies pick the next token. We will carry our consistent toy example through two full generation steps, showing concrete numbers at every stage.

---

What Are Logits?

A logit is an unnormalized numerical score produced by the model’s final linear projection for each token in the vocabulary. The model outputs one logit per vocabulary item — in our toy model, 7 numbers.

Key idea: A logit is not a probability. It can be negative, positive, or zero. Higher logit = the model believes this token is more likely to come next. But the raw values don’t sum to 1 and can’t be interpreted as chances.

Where do logits come from? The transformer’s final hidden state for the last position (a vector of size $d$, e.g., 4 in our toy model) is multiplied by the output projection matrix (also called the unembedding matrix or LM head) of shape [vocab_size, d]. This produces a vector of length $V = 7$ — one score per token.

Vocabulary index	Token	Logit (illustrative)
0	[PAD]	-3.2
1	The	0.8
2	cat	1.2
3	sat	0.5
4	dog	-0.4
5	on	2.1
6	mat	0.9

Note: These logits are illustrative values chosen to demonstrate the mechanics. They are not from a trained model.

---

From Logits to Probabilities: The Softmax Function

To turn logits into valid probabilities (all positive, summing to exactly 1.0), we apply the softmax function:

softmax(z_i) = exp(z_i) / sum(exp(z_j) for j in 1..V)

Where:

• z_i is the logit for token i
• exp(z_i) is the exponential (always positive)
• The denominator sums exponentials over all vocabulary items, guaranteeing the outputs sum to 1

Step-by-Step Softmax Calculation (Illustrative Logits)

Using the logits above:

Token ID	Token	Logit $z_i$	exp($z_i$) (rounded)
0	[PAD]	-3.2	0.0408
1	The	0.8	2.2255
2	cat	1.2	3.3201
3	sat	0.5	1.6487
4	dog	-0.4	0.6703
5	on	2.1	8.1662
6	mat	0.9	2.4596

Sum of exponentials: $0.0408 + 2.2255 + 3.3201 + 1.6487 + 0.6703 + 8.1662 + 2.4596 =$ 18.5312

Now divide each exponential by the sum:

Token ID	Token	Probability $p_i$ (rounded)
0	[PAD]	0.0022
1	The	0.1201
2	cat	0.1792
3	sat	0.0890
4	dog	0.0362
5	on	0.4407
6	mat	0.1327

Check: $0.0022 + 0.1201 + 0.1792 + 0.0890 + 0.0362 + 0.4407 + 0.1327 =$ 1.0001 (rounding)

The highest probability is token 5 (on) at ~44%.

---

Generation Step 1: From Prompt to First Generated Token

Prompt (from Lesson 2)

"The cat sat"
Token IDs: [1, 2, 3]

Autoregressive Loop — Step 1

1. Input: Token IDs [1, 2, 3] → Embedding lookup → Tensor [1, 3, 4]
2. Forward pass: Transformer processes all 3 positions in parallel (prefill)
3. Last position output: Hidden state for position 3 (sat) → Output projection → Logits vector of length 7
4. Softmax: Convert logits → Probabilities (table above)
5. Decoding strategy: Choose next token ID

Greedy Decoding (Argmax)

Greedy decoding (also called argmax decoding) always selects the single token with the highest probability.

Greedy decoding = deterministic. Same prompt, same model → same output every time.

From our probabilities, on (ID 5, probability 0.4407) wins.

Generated so far: "The cat sat on" — Token IDs [1, 2, 3, 5]

---

Generation Step 2: The Autoregressive Loop Continues

Now the model runs again with the extended sequence including the newly generated token.

Step 2 Input

"The cat sat on"
Token IDs: [1, 2, 3, 5]

The transformer processes all 4 positions again (prefill for the first 3, decode for the new position). The last position is now on (ID 5).

Step 2 Illustrative Logits

Token ID	Token	Logit (illustrative)
0	[PAD]	-4.0
1	The	0.3
2	cat	0.1
3	sat	-0.2
4	dog	-1.5
5	on	0.2
6	mat	2.5

Step 2 Softmax

Token ID	Token	Logit $z_i$	exp($z_i$)	Probability $p_i$
0	[PAD]	-4.0	0.0183	0.0011
1	The	0.3	1.3499	0.0798
2	cat	0.1	1.1052	0.0653
3	sat	-0.2	0.8187	0.0484
4	dog	-1.5	0.2231	0.0132
5	on	0.2	1.2214	0.0722
6	mat	2.5	12.1825	0.7200

Sum of exponentials = 16.9191

Greedy pick: mat (ID 6, probability 0.7200)

Generated so far: "The cat sat on mat" — Token IDs [1, 2, 3, 5, 6]

---

The Autoregressive Generation Loop (Summary)

START: Prompt tokens [1, 2, 3]  ("The cat sat")

LOOP:
  1. Run forward pass on ALL tokens so far
  2. Extract logits for LAST position only
  3. Apply softmax → probability distribution over V
  4. Apply DECODING STRATEGY → choose next token ID
  5. Append chosen ID to sequence
  6. Check STOPPING CONDITIONS (EOS or max tokens)
  7. If not stopped → repeat LOOP
END: Full generated sequence

After Step 2, if the model outputs an end-of-sequence token (e.g., [EOS] / <|endoftext|>) or we hit a maximum token limit, generation stops.

---

Beyond Greedy: Stochastic Decoding Strategies

Greedy decoding is fast and deterministic but often produces repetitive, generic, or “loopy” text. Stochastic (randomized) sampling introduces variety by drawing from the probability distribution.

Temperature Scaling

Temperature ($T$) scales logits before softmax:

p_i = exp(z_i / T) / sum(exp(z_j / T) for all j)

• T = 1: Original probabilities (no change)
• T < 1 (e.g., 0.5): Sharpens differences — high-probability tokens become even more dominant; approaches greedy as T → 0
• T > 1 (e.g., 2.0): Flattens differences — low-probability tokens get relatively more mass; increases variety

Origin: Temperature comes from statistical physics (Boltzmann distribution) and was adapted for neural language models (Ackley, Hinton, & Sejnowski, 1985).

Temperature Effect Example (Step 1 Logits)

Token	Logit	$T=1$ (base)	$T=0.5$ (sharp)	$T=2.0$ (flat)
on	2.1	0.441	0.726	0.284
cat	1.2	0.179	0.120	0.181
mat	0.9	0.133	0.066	0.156
The	0.8	0.120	0.054	0.149
sat	0.5	0.089	0.030	0.128
dog	-0.4	0.036	0.005	0.082
[PAD]	-3.2	0.002	0.000	0.020

Observation: At $T=0.5$, on jumps to 73% (near-greedy). At $T=2.0$, on drops to 28% and others rise — more variety, more risk of nonsense.

---

Top-$k$ Sampling

Top-$k$ sampling keeps only the $k$ highest-probability tokens, sets all others to zero, and renormalizes.

Effect: Hard cutoff. If $k=3$, only the top 3 tokens can ever be sampled, regardless of how small their probabilities are.

Using Step 1 probabilities ($T=1$), top-3 tokens:

1. on (0.441)
2. cat (0.179)
3. mat (0.133)

Tokens The, sat, dog, [PAD] → probability set to 0. Renormalize over the 3 kept:

• Sum kept = 0.441 + 0.179 + 0.133 = 0.753
• New probabilities: on = 0.441/0.753 = 0.586, cat = 0.238, mat = 0.177

---

Top-$p$ (Nucleus) Sampling

Top-$p$ sampling (nucleus sampling; Holtzman et al., 2019) keeps the smallest set of top tokens whose cumulative probability $\geq p$.

Effect: Adaptive cutoff. When the model is confident (sharp distribution), fewer tokens qualify. When uncertain (flat distribution), more tokens qualify.

Using Step 1 probabilities ($T=1$), cumulative sorted:

Rank	Token	Probability	Cumulative
1	on	0.441	0.441
2	cat	0.179	0.620
3	mat	0.133	0.753
4	The	0.120	0.873
5	sat	0.089	0.962
6	dog	0.036	0.998
7	[PAD]	0.002	1.000

• $p = 0.70$: Keep tokens until cumulative ≥ 0.70 → keep on, cat, mat (cumulative 0.753). Same as top-3 here.
• $p = 0.90$: Keep on, cat, mat, The (cumulative 0.873) + sat (cumulative 0.962) → top 5 tokens
• $p = 0.95$: Keep top 5 (cumulative 0.962)

Then renormalize over the kept set.

---

Repetition Penalties

Without intervention, greedy or high-temperature sampling can loop: "The cat sat on the mat on the mat on the mat..."

Repetition penalty reduces the logit of any token that has already appeared in the prompt or recent generation (Keskar et al., 2019).

Common formula (multiplicative):

• If token $i$ has appeared: $z_i \leftarrow z_i / \alpha$ (for $\alpha > 1$)
• If token $i$ has not appeared: $z_i$ unchanged

Typical $\alpha = 1.1$ to $1.2$.

Example (Step 2 with Repetition Penalty $\alpha=1.2$)

Step 2 logits before penalty:

Token	Logit
on	0.2
mat	2.5

on (ID 5) already appeared in Step 1 generation. Apply penalty:

• Adjusted logit for on: $0.2 / 1.2 = 0.1667$
• Adjusted logit for mat: unchanged $= 2.5$

Result: mat becomes even more dominant. The penalty discourages immediately repeating on.

---

Stopping Rules

Generation halts when either condition is met:

1. End-of-sequence token ([EOS] / <|endoftext|>) is generated. The model learns during training to emit this token when a natural stopping point is reached.
2. Maximum token limit is reached (e.g., max_new_tokens = 100). A hard safety ceiling to prevent infinite loops.

Practical note: Production systems also implement stop sequences (specific strings like "\n\n" or "</s>") that trigger termination even if not exactly the trained EOS token.

---

Worked Exercises & Solutions

Exercise 1: Hand-Calculate Softmax

Problem: Compute softmax probabilities for these 3 logits: [2.0, 1.0, 0.0]. Show exponentials, sum, and final probabilities. Verify they sum to 1.0.

Solution:

Index	Logit $z_i$	exp($z_i$)
0	2.0	7.3891
1	1.0	2.7183
2	0.0	1.0000

Sum = $7.3891 + 2.7183 + 1.0000 =$ 11.1074

Probabilities:

• $p_0 = 7.3891 / 11.1074 =$ 0.6652
• $p_1 = 2.7183 / 11.1074 =$ 0.2447
• $p_2 = 1.0000 / 11.1074 =$ 0.0900

Check: $0.6652 + 0.2447 + 0.0900 =$ 0.9999 ✓ (rounding)

---

Exercise 2: Top-$k$ and Top-$p$ Filtering

Problem: Given this probability distribution over 6 tokens:

Token	A	B	C	D	E	F
Prob	0.50	0.20	0.15	0.10	0.04	0.01

1. Apply top-$k=3$. List kept tokens and their renormalized probabilities.
2. Apply top-$p=0.80$. List kept tokens and their renormalized probabilities.

Solution:

1. Top-$k=3$: Keep A (0.50), B (0.20), C (0.15). Sum = 0.85. Renormalized:

• A: $0.50 / 0.85 =$ 0.588
• B: $0.20 / 0.85 =$ 0.235
• C: $0.15 / 0.85 =$ 0.176

2. Top-$p=0.80$: Cumulative: A=0.50, A+B=0.70, A+B+C=0.85 ≥ 0.80. Keep A, B, C (same as top-3 here). Sum = 0.85. Renormalized: same as above (0.588, 0.235, 0.176).

Note: With a flatter distribution, top-$p$ would keep more tokens than a fixed $k$.

---

Exercise 3: Temperature Effect Calculation

Problem: For logits [3.0, 1.0, 0.0], compute probabilities at $T=0.5$, $T=1.0$, and $T=2.0$. Comment on how the highest-probability token changes.

Solution:

Token	Logit	$T=0.5$ (logit/0.5)	$T=1.0$	$T=2.0$ (logit/2.0)
0	3.0	6.0	3.0	1.5
1	1.0	2.0	1.0	0.5
2	0.0	0.0	0.0	0.0

$T=0.5$: Exps = [403.43, 7.39, 1.00], Sum = 411.82

• $p_0 =$ 0.9796, $p_1 = 0.0179$, $p_2 = 0.0024$

$T=1.0$: Exps = [20.09, 2.72, 1.00], Sum = 23.81

• $p_0 =$ 0.8437, $p_1 = 0.1142$, $p_2 = 0.0420$

$T=2.0$: Exps = [4.48, 1.65, 1.00], Sum = 7.13

• $p_0 =$ 0.6285, $p_1 = 0.2312$, $p_2 = 0.1403$

Observation: Token 0 probability drops from 98% ($T=0.5$) → 84% ($T=1$) → 63% ($T=2$). Lower temperature = more confident/peaky. Higher temperature = more spread/diverse.

---

Primary Source Citations

• Logits & output projection: Vaswani et al. (2017), Attention Is All You Need, NeurIPS 2017 (arXiv:1706.03762) — Linear projection from hidden state to vocabulary logits.
• Temperature scaling foundation: Ackley, Hinton, & Sejnowski (1985), A Learning Algorithm for Boltzmann Machines, Cognitive Science (PNAS) — Boltzmann distribution / temperature in probabilistic models.
• Top-$p$ (nucleus) sampling: Holtzman et al. (2019), The Curious Case of Neural Text Degeneration, ICLR 2020 (arXiv:1904.09751).
• Repetition penalty: Keskar et al. (2019), CTRL: A Conditional Transformer Language Model for Controllable Generation (arXiv:1909.05858).

---

Summary

Concept	What It Does	Key Formula / Rule
Logits	Unnormalized scores per vocabulary token	Output projection × last hidden state
Softmax	Converts logits → probabilities summing to 1	p_i = exp(z_i) / sum(exp(z_j))
Greedy (argmax)	Picks highest-probability token	argmax_i p_i
Temperature ($T$)	Scales logits before softmax; T < 1 sharpens, T > 1 flattens	p_i = exp(z_i / T) / sum(exp(z_j / T))
Top-$k$	Keeps only top $k$ tokens, zeroes rest, renormalizes	Hard cutoff at rank $k$
Top-$p$ (nucleus)	Keeps smallest top set with cumulative prob $\ge p$	Adaptive cutoff
Repetition penalty	Divides logits of seen tokens by $\alpha > 1$	z_i ← z_i / α
Stopping	EOS token or max token limit reached	—

---

What Comes Next

• Next lesson: Vectors, Matrices, and Tensor Shapes — Build the linear-algebra toolkit used throughout the transformer.
• Previous lesson: From Raw Text to Token IDs and Embeddings
• Track overview: Start Here: LLM Foundations