Attention from Numbers: Queries, Keys, and Values

In Token Embeddings and Position in a Decoder, you learned how token embeddings enter the model and how RoPE rotates Queries and Keys to encode relative position. Now we open the attention mechanism itself. We will define every symbol in the scaled dot-product attention formula, explain the physical intuition behind Queries, Keys, and Values, prove why the scaling factor sqrt(d_k) is mathematically necessary, and work through a complete numerical example from raw numbers to the final output representations.

---

The High-Level Analogy: Retrieval by Meaning

Before any formulas, imagine a database lookup:

• Query (Q) — “What am I looking for?” The active token’s search intent.
• Key (K) — “What information do I have?” Each context token’s catalog entry.
• Value (V) — “What content should be retrieved?” The actual semantic content passed forward.
• Similarity Q · K — How relevant is this key to my query?
• Softmax — Converts similarity scores into percentage weights that sum to 100%.
• Attention Output — A weighted blend of Value vectors according to those percentage weights.

Every token in the sequence simultaneously acts as a Query (searching) and as a Key/Value pair (being searched). The attention mechanism computes all pairwise similarities at once.

---

The Core Mathematical Formula

Scaled dot-product attention is defined as:

Attention(Q, K, V) = softmax( (Q @ K^T) / sqrt(d_k) ) @ V

Let us define every component with its tensor shape:

Symbol	Shape	Meaning
Q	[T, d_k]	Query matrix — one row per token
K	[T, d_k]	Key matrix — one row per token
K^T	[d_k, T]	Transpose of K
Q @ K^T	[T, T]	Raw attention scores — each entry S_ij = q_i · k_j
sqrt(d_k)	scalar	Scaling factor (square root of head dimension)
softmax(…)	[T, T]	Row-wise softmax — each row sums to 1.0
V	[T, d_v]	Value matrix — one row per token
Output	[T, d_v]	Context-mixed representations for each token

• T = sequence length (number of tokens)
• d_k = Key/Query head dimension
• d_v = Value head dimension (often d_v = d_k)

Row-wise softmax means: for each query token i, we take the row S_i = [S_i0, S_i1, ..., S_i(T-1)], exponentiate every element, divide by the row sum, producing probabilities A_i = [A_i0, ..., A_i(T-1)] with sum_j A_ij = 1. The output for token i is sum_j A_ij * v_j — a convex combination of all Value vectors.

---

Why Divide by sqrt(d_k)? (The Scaling Factor)

This is not a heuristic. It follows directly from the statistics of dot products.

Suppose q and k are independent random vectors of dimension d_k, each element with mean 0 and variance 1. Their dot product is:

q · k = sum_{i=1 to d_k} q_i * k_i

Each product q_i * k_i has mean 0 and variance 1 (since Var(XY) = E[X]^2 Var(Y) + E[Y]^2 Var(X) + Var(X)Var(Y) = 1 when means are 0 and variances are 1). The sum of d_k independent such terms has variance d_k and standard deviation sqrt(d_k).

For typical head dimensions:

• d_k = 64 → standard deviation = 8
• d_k = 128 → standard deviation ≈ 11.3

Without scaling, raw scores S_ij can easily reach ±20 or ±30. When these enter softmax:

exp(30) ≈ 1.06 × 10^13
exp(10) ≈ 22,026

The ratio is billions to one. Softmax saturates: one token receives weight ≈ 0.9999…, all others ≈ 0. Gradients vanish. Training stalls.

Dividing by sqrt(d_k) scales the variance back to 1.0, keeping scores in a healthy range where softmax produces meaningful gradients and all tokens can contribute.

---

How Q, K, and V Are Produced from Input Tokens

The input to an attention layer is X ∈ ℝ^{T × d_model}.

Q = X @ W_q    where W_q ∈ ℝ^{d_model × d_k}
K = X @ W_k    where W_k ∈ ℝ^{d_model × d_k}
V = X @ W_v    where W_v ∈ ℝ^{d_model × d_v}

After attention computes O ∈ ℝ^{T × d_v}, an output projection maps it back to model dimension:

Output = O @ W_o    where W_o ∈ ℝ^{d_v × d_model}

In multi-head attention, this happens in parallel for each head with separate projection matrices, then the head outputs are concatenated and projected once more. For this lesson we focus on a single head.

---

Complete Worked Numerical Example (T=2, d_k=4, d_v=2)

We use d_k = 4 so sqrt(d_k) = sqrt(4) = 2.0 exactly — clean arithmetic throughout.

Given Input Matrices

Q = [[2.0, 0.0, 2.0, 0.0],
     [0.0, 4.0, 0.0, 2.0]]     shape [2, 4]

K = [[1.0, 0.0, 1.0, 0.0],
     [0.0, 1.0, 0.0, 0.0]]     shape [2, 4]

V = [[1.0, 2.0],
     [3.0, 0.0]]               shape [2, 2]

Step 1: Compute K^T (shape [4, 2])

K^T = [[1.0, 0.0],
       [0.0, 1.0],
       [1.0, 0.0],
       [0.0, 0.0]]

Column 0 of K^T is token 0’s key k_0 = [1.0, 0.0, 1.0, 0.0]^T; column 1 is token 1’s key k_1 = [0.0, 1.0, 0.0, 0.0]^T. When multiplying Q @ K^T, row i of Q dots with column j of K^T to compute q_i · k_j.

Step 2: Compute Raw Scores S_raw = Q @ K^T (shape [2, 2])

S_00 = q_0 · k_0 = 2(1) + 0(0) + 2(1) + 0(0) = 4.0
S_01 = q_0 · k_1 = 2(0) + 0(1) + 2(0) + 0(0) = 0.0
S_10 = q_1 · k_0 = 0(1) + 4(0) + 0(1) + 2(0) = 0.0
S_11 = q_1 · k_1 = 0(0) + 4(1) + 0(0) + 2(0) = 4.0

S_raw = [[4.0, 0.0],
         [0.0, 4.0]]

Step 3: Scale by sqrt(d_k) = 2.0

S = S_raw / 2.0 = [[2.0, 0.0],
                    [0.0, 2.0]]

Step 4: Row-wise Softmax

Row 0: [2.0, 0.0]

exp(2.0) = 7.389056...
exp(0.0) = 1.0
sum = 8.389056...

A_00 = 7.389056 / 8.389056 = 0.880797...
A_01 = 1.000000 / 8.389056 = 0.119202...

Row 1: [0.0, 2.0]

exp(0.0) = 1.0
exp(2.0) = 7.389056...
sum = 8.389056...

A_10 = 1.000000 / 8.389056 = 0.119202...
A_11 = 7.389056 / 8.389056 = 0.880797...

Attention Matrix A:

A = [[0.8808, 0.1192],
     [0.1192, 0.8808]]

(Values rounded to 4 decimal places for readability; full precision used in next step.)

Step 5: Multiply by V (A @ V) → Output shape [2, 2]

V = [[1.0, 2.0],
     [3.0, 0.0]]

Output Row 0:

0.880797 * [1.0, 2.0] + 0.119202 * [3.0, 0.0]
= [0.880797 + 0.357607, 1.761594 + 0.0]
= [1.238404, 1.761594]

Output Row 1:

0.119202 * [1.0, 2.0] + 0.880797 * [3.0, 0.0]
= [0.119202 + 2.642392, 0.238405 + 0.0]
= [2.761594, 0.238405]

Final Output Matrix:

Output = [[1.2384, 1.7616],
          [2.7616, 0.2384]]    shape [2, 2]

Interpretation

• Token 0 attended 88.1% to itself and 11.9% to Token 1. Its output blends mostly its own Value vector [1.0, 2.0] with a small contribution from Token 1’s Value [3.0, 0.0].
• Token 1 did the opposite: 11.9% to Token 0, 88.1% to itself.

The attention mechanism has computed a context-weighted blend of Value vectors for each token.

---

Hands-On Exercises with Checked Solutions

Exercise 1: The Variance-Scaling Effect

Problem: Two vectors q, k of dimension d_k = 64 have dot product q · k = 16.0. Calculate the scaled dot product and compare the exponential magnitudes.

Solution:

Step 1: Compute scaling factor

sqrt(d_k) = sqrt(64) = 8.0

Step 2: Apply scaling

Scaled score = 16.0 / 8.0 = 2.0

Step 3: Compare exponentials

exp(16.0) ≈ 8,886,110
exp(2.0)  ≈ 7.389

Explanation: Without scaling, a score of 16 would make softmax essentially a one-hot vector (weight ≈ 0.999999). With scaling, score 2.0 gives exp(2)/(exp(2)+exp(other)) — healthy gradients, all tokens can contribute. The scaling factor compresses the dynamic range by a factor of sqrt(d_k).

---

Exercise 2: Single-Query Attention Over 2 Keys

Problem: q = [1.0, 2.0], k_0 = [2.0, 1.0], k_1 = [0.0, 2.0], d_k = 4 (so sqrt(d_k) = 2.0). Values: v_0 = [10.0, 0.0], v_1 = [0.0, 20.0]. Compute the attention output.

Solution:

Step 1: Raw dot products

q · k_0 = 1(2) + 2(1) = 4.0
q · k_1 = 1(0) + 2(2) = 4.0

Step 2: Scale by sqrt(4) = 2.0

Scaled: 4.0 / 2.0 = 2.0  (both keys)

Step 3: Softmax (both scores equal → uniform)

p = [0.50, 0.50]

Step 4: Weighted sum of Values

Output = 0.50 * [10, 0] + 0.50 * [0, 20]
       = [5.0, 10.0]

The query treats both keys as equally relevant and retrieves an even blend of their Values.

---

Exercise 3: Full Tensor Shape Flow

Problem: Given batch size B=1, sequence length T=4, model dim d_model=8, head dim d_k=d_v=4. State exact shapes of every tensor.

Solution:

Stage	Tensor	Shape	Notes
Input	X	[1, 4, 8]	(B, T, d_model)
Projection	W_q	[8, 4]	(d_model, d_k)
Query	Q = X @ W_q	[1, 4, 4]	(B, T, d_k)
Key	K	[1, 4, 4]	Same as Q
Key Transpose	K^T	[1, 4, 4]	Last two dims transposed
Scores	Q @ K^T	[1, 4, 4]	(B, T, T)
Scaled	/ sqrt(d_k)	[1, 4, 4]	Elementwise
Softmax	A	[1, 4, 4]	Row-wise on last dim
Value	V	[1, 4, 4]	(B, T, d_v)
Output	A @ V	[1, 4, 4]	(B, T, d_v)
Out Proj	W_o	[4, 8]	(d_v, d_model)
Final	Output @ W_o	[1, 4, 8]	Back to model dim

---

Summary Reference Table

Stage	Tensor	Shape (T=2, d_k=4, d_v=2)	Formula	Physical Purpose
Input	X	[2, 8]	—	Token representations
Project	W_q	[8, 4]	—	Learned: “what to search for”
Query	Q	[2, 4]	X @ W_q	Active search vectors
Project	W_k	[8, 4]	—	Learned: “what I offer”
Key	K	[2, 4]	X @ W_k	Catalog entries for matching
Transpose	K^T	[4, 2]	K.transpose()	Aligns for dot product
Scores	S_raw	[2, 2]	Q @ K^T	Pairwise relevance (unnormalized)
Scale	S	[2, 2]	S_raw / sqrt(d_k)	Prevents softmax saturation
Weights	A	[2, 2]	softmax(S)	Percentage allocation per query
Project	W_v	[8, 2]	—	Learned: “what content to pass”
Value	V	[2, 2]	X @ W_v	Content vectors to blend
Output	O	[2, 2]	A @ V	Context-mixed representations
Out Proj	W_o	[2, 8]	—	Maps back to model dimension

---

Primary Source Citations

• Vaswani et al. (2017), Attention Is All You Need, Section 3.2.1 — arXiv:1706.03762

---

What Comes Next

• Next lesson: Causal Masks and Attention Heads — Apply a causal mask and compare multi-head, multi-query, and grouped-query attention using explicit head and cache shapes.
• Previous lesson: Token Embeddings and Position in a Decoder
• Track overview: Start Here: LLM Foundations