Token Embeddings and Position in a Decoder

In Normalization, Computation Cost, and Number Formats, you learned how RMSNorm keeps activations stable across layers, why attention’s quadratic complexity dominates at long contexts, and how number formats like BF16 and INT4 determine whether a model fits in your GPU memory. That lesson concluded Part II, our mathematical foundations.

Now we begin Part III: Inside a Decoder-Only Transformer. In this part, we trace the forward pass of a decoder-only model from input tokens all the way to output logits. This first lesson in Part III answers a fundamental question: how does a transformer know which word comes first?

---

The Problem: Attention Is Word-Order Blind

In From Raw Text to Token IDs and Embeddings, we converted tokens like "cat" into embedding vectors. Each token gets a vector of numbers — its token embedding — looked up from a table. So far, so good.

Now imagine we feed two sentences into a transformer:

1. "The dog chased the cat"
2. "The cat chased the dog"

The words are identical. The token embeddings are identical. The only difference is order.

Self-attention computes dot products between token vectors. If you swap the order of words, the set of dot products between word pairs stays exactly the same! The attention mechanism by itself has no concept of “first”, “second”, or “before”. It treats the input as a bag of tokens.

This property has a name: permutation equivariance. If you permute (reorder) the input tokens, the output tokens are permuted in exactly the same way. The model cannot distinguish word order from the attention mechanism alone.

Without explicit positional information, a transformer is word-order blind.

---

The Evolution of Position in Transformers

1. Absolute Positional Encodings (Original Transformer, GPT-2)

The original Transformer (Vaswani et al., 2017) and GPT-2 (Radford et al., 2019) solved this by adding a position vector to each token embedding:

x_t = e_t + p_t

Where:

• e_t = token embedding at position t
• p_t = position embedding (learned or sinusoidal) for position t
• x_t = combined input to the first transformer layer

Sinusoidal encodings (from the original paper) use fixed sine and cosine functions at different frequencies so the model can extrapolate to positions it hasn’t seen. Learned absolute embeddings (GPT-2) simply learn a separate embedding vector for each position index 0, 1, 2, ..., max_length-1.

Limitations of absolute encodings:

• Cannot generalize well beyond the maximum sequence length seen during training (especially learned embeddings)
• Do not explicitly represent the relative distance |i - j| between two tokens — the model must learn this from the absolute positions
• The position signal is baked in at the very bottom and cannot adapt per layer

---

2. Relative Positional Approaches

Before RoPE, several methods tried to inject relative distance directly into attention:

• Shaw et al. (2018) added learnable relative position biases to the attention scores
• T5 (Raffel et al., 2020) used relative position buckets with shared biases
• ALiBi (Press et al., 2021) added a fixed linear bias that penalizes attention to distant tokens

These approaches modify the attention score computation. They work, but they add parameters or change the attention formula in ways that don’t compose cleanly with other optimizations.

---

3. Rotary Position Embedding — RoPE (Su et al., 2021)

RoPE is the dominant method in modern open-weight LLMs — Llama 2/3, Mistral, Gemma, Qwen, DeepSeek, and many others all use RoPE.

The key insight: instead of adding a position vector at the input, RoPE rotates the Query and Key vectors in 2D planes after the linear projection inside every attention layer.

This means:

• Position information is applied fresh at every layer, not just once at the bottom
• The rotation encodes relative distance directly into the dot product Q @ K^T
• No extra parameters are needed — the rotation frequencies are fixed by a simple formula
• The vector norm is preserved (rotation doesn’t change length)

---

How RoPE Works: 2D Plane Rotations

The Core Idea

Take a Query or Key head vector of dimension d_head (for example, d_head = 128 in Llama-2-7B).

Group the coordinates into d_head / 2 pairs:

(x_0, x_1), (x_2, x_3), ..., (x_{d-2}, x_{d-1})

Each pair i gets a base angular frequency:

θ_i = 10000^(-2i / d_head)

For a token at position index m (where m = 0, 1, 2, ...), rotate each 2D pair by angle m * θ_i:

[ x_new_0 ]   [ cos(m*θ_i)  -sin(m*θ_i) ] [ x_0 ]
[ x_new_1 ] = [ sin(m*θ_i)   cos(m*θ_i) ] [ x_1 ]

In formulas:

x_new_0 = x_0 * cos(m * θ_i) - x_1 * sin(m * θ_i)
x_new_1 = x_0 * sin(m * θ_i) + x_1 * cos(m * θ_i)

This is a standard 2D rotation matrix. Rotating a vector preserves its length (the Euclidean norm is unchanged).

Why Different Frequencies for Different Pairs?

Each pair i rotates at a different speed. Low-index pairs (early dimensions) rotate slowly — they capture long-range position patterns. High-index pairs rotate quickly — they capture fine-grained, local position patterns.

This is analogous to how the original sinusoidal encodings used different frequencies, but here the frequencies are applied as rotations in the Query/Key space itself.

---

The Relative Distance Magic

Here is the crucial property. When Query q_m at position m is dotted with Key k_n at position n:

The dot product contains terms like:

cos(m * θ_i) * cos(n * θ_i) + sin(m * θ_i) * sin(n * θ_i)

Using the cosine angle subtraction identity:

cos(A - B) = cos(A)cos(B) + sin(A)sin(B)

This simplifies to:

cos((m - n) * θ_i)

The attention score depends ONLY on the relative distance m - n, not on the absolute positions m or n!

This is exactly what we want: whether “dog” appears at position 2 or position 100, its relationship to “chased” at position 3 or 101 should depend only on their separation (1 step apart), not their absolute locations.

---

Why Apply RoPE to Q and K, but NOT V?

• Query (Q) and Key (K) determine where to pay attention (the attention weights). Relative position is critical for determining which words relate to which.
• Value (V) represents what content is retrieved and passed forward. Content should not be rotated or distorted by position — the meaning of “dog” shouldn’t change based on where it appears.

RoPE is applied inside each attention layer after the W_q and W_k projections, before computing Q @ K^T.

---

Hands-On Exercises & Checked Solutions

Exercise 1: 2D RoPE Rotation by Hand

Problem: Rotate vector v = [1.0, 0.0] at position m = 1 with angle θ = π/4 = 45°.

Given:

• cos(45°) ≈ 0.7071
• sin(45°) ≈ 0.7071

Solution:

Step 1: Apply the rotation formulas

x_new_0 = 1.0 * cos(45°) - 0.0 * sin(45°) = 1.0 * 0.7071 - 0 = 0.7071
x_new_1 = 1.0 * sin(45°) + 0.0 * cos(45°) = 1.0 * 0.7071 + 0 = 0.7071

Rotated vector: v' = [0.7071, 0.7071]

Step 2: Verify norm preservation

Original norm: sqrt(1.0^2 + 0.0^2) = 1.0
New norm:      sqrt(0.7071^2 + 0.7071^2) = sqrt(0.5 + 0.5) = sqrt(1.0) = 1.0000

The norm is perfectly preserved (up to floating-point rounding).

---

Exercise 2: Relative Distance Property Verification

Problem: Query q = [1.0, 0.0] at position m = 2, Key k = [1.0, 0.0] at position n = 1. Angle θ = 45°. Verify the dot product equals cos((m - n) * θ).

Solution:

Step 1: Compute relative distance

m - n = 2 - 1 = 1 step (45°)

Step 2: Rotate Query at position m = 2 Angle = 2 * 45° = 90°

cos(90°) = 0,  sin(90°) = 1
q' = [1.0 * 0 - 0.0 * 1, 1.0 * 1 + 0.0 * 0] = [0.0, 1.0]

Step 3: Rotate Key at position n = 1 Angle = 1 * 45° = 45°

cos(45°) ≈ 0.7071,  sin(45°) ≈ 0.7071
k' = [1.0 * 0.7071 - 0.0 * 0.7071, 1.0 * 0.7071 + 0.0 * 0.7071] = [0.7071, 0.7071]

Step 4: Dot product of rotated vectors

q' · k' = (0.0 * 0.7071) + (1.0 * 0.7071) = 0.7071

Step 5: Verify with cosine of relative difference

cos((m - n) * θ) = cos(1 * 45°) = cos(45°) = 0.7071

Match! The dot product equals the cosine of the relative difference.

---

Exercise 3: Higher Position and Norm Invariance

Problem: Rotate vector w = [3.0, 4.0] (norm 5.0) at position m = 2 with θ = 90° (so rotation angle = 2 * 90° = 180°).

Given: cos(180°) = -1, sin(180°) = 0

Solution:

Step 1: Apply rotation

w'_0 = 3.0 * (-1) - 4.0 * 0 = -3.0
w'_1 = 3.0 * 0 + 4.0 * (-1) = -4.0
Rotated: w' = [-3.0, -4.0]

Step 2: Verify norm preservation

Original norm: sqrt(3.0^2 + 4.0^2) = sqrt(9 + 16) = sqrt(25) = 5.0
New norm:      sqrt((-3.0)^2 + (-4.0)^2) = sqrt(9 + 16) = sqrt(25) = 5.0

Exact preservation. A 180° rotation flips the vector to the opposite direction but keeps its length exactly the same.

---

Summary Reference Table

Aspect	Absolute Positional Encodings	RoPE (Rotary Position Embedding)
Where applied	Added to input embeddings once at bottom	Applied to Q and K inside every attention layer
Mechanism	Vector addition: x = e + p	2D rotation in Q/K space
Position type	Absolute (position index t)	Relative (distance m - n encoded in dot product)
Generalization	Poor beyond max training length (especially learned)	Extrapolates naturally; base 10000 gives smooth scaling
Parameters	Learned: max_len × d_model params	Zero — frequencies fixed by formula
Norm preservation	N/A (addition changes norm)	Yes — rotation preserves vector norm
Attention formula	Unchanged (standard Q @ K^T)	Unchanged — RoPE is pre-processing on Q/K
Used by	Original Transformer, GPT-2, BERT	Llama 2/3, Mistral, Gemma, Qwen, DeepSeek, Phi, etc.

---

Primary Source Citations

• RoPE (RoFormer): Su et al. (2021), RoFormer: Enhanced Transformer with Rotary Position Embedding — arXiv:2104.09864
• Original Transformer (absolute sinusoidal): Vaswani et al. (2017), Attention Is All You Need, Section 3.5 — arXiv:1706.03762
• Llama 2 (RoPE adoption): Touvron et al. (2023), Llama 2: Open Foundation and Fine-Tuned Chat Models — arXiv:2307.09288
• GPT-2 (learned absolute): Radford et al. (2019), Language Models are Unsupervised Multitask Learners — OpenAI Blog

---

What Comes Next

• Next lesson: Attention from Numbers: Queries, Keys, and Values — Derive scaled dot-product attention, define every symbol and shape, and calculate a complete tiny Q/K/V example.
• Previous lesson: Normalization, Computation Cost, and Number Formats
• Track overview: Start Here: LLM Foundations