Linear Projections, Transposes, and Similarity

In Vectors, Matrices, and Tensor Shapes, you learned how matrix multiplication works and how to track tensor shapes through the embedding and output-projection stages of a transformer. This lesson introduces three more tools you will see in every attention block:

• Linear projections — the Q, K, V weight matrices that transform token representations
• Matrix transpose — flipping rows and columns so shapes line up for Q @ K^T
• Vector norms and cosine similarity — measuring how aligned two vectors are, independent of their lengths

By the end, you will trace shapes from an input tensor through Q and K projections to the square attention-score matrix, and you will compute a cosine similarity by hand.

---

Linear Projections: Mapping Between Feature Spaces

A linear projection multiplies an input vector (or tensor) by a learned weight matrix W and optionally adds a bias vector b. It rotates, scales, or changes the dimension of the representation.

In math notation:

y = x @ W^T + b        (input x is a row vector)

Or equivalently for column vectors:

y = W @ x + b

In deep learning frameworks, the weight matrix is usually stored with shape [d_out, d_in]. When the input is a batch of row vectors with shape [..., d_in], the operation is:

output = input @ W^T      # [..., d_in] @ [d_in, d_out] = [..., d_out]

Why Project?

Token embeddings live in a general-purpose feature space (e.g., d = 4 in our toy model). Attention needs three different views of each token:

• Query (Q): “What am I looking for?”
• Key (K): “What do I contain?”
• Value (V): “What do I pass along if attended to?”

Each view is a linear projection with its own learned matrix:

Q = X @ W_q^T
K = X @ W_k^T
V = X @ W_v^T

All three matrices W_q, W_k, W_v have shape [d_head, d_model] (or [d_model, d_model] if using the full model dimension). The projection changes the representation so each head can specialize.

Worked Example: Small Projection

Input tensor X has shape [1, 3, 4] (batch=1, seq_len=3, d_model=4). We project to a head dimension d_head = 3.

Weight matrix W_q (query projection): shape [3, 4]

W_q = [ 0.5  -0.2   0.1   0.0 ]
      [ 0.0   0.4  -0.3   0.2 ]
      [ 0.1   0.1   0.5  -0.1 ]

Shape trace:

• X: [1, 3, 4]
• W_q^T: [4, 3] (transpose of [3, 4])
• Q = X @ W_q^T: [1, 3, 4] @ [4, 3] → inner dim 4 matches ✓ → result [1, 3, 3]

The query matrix Q now has shape [1, 3, 3] — three tokens, each with a 3-dimensional query vector.

Key idea: A linear projection keeps the batch and sequence dimensions intact. Only the last dimension changes (from d_in to d_out).

---

Matrix Transpose: Flipping Rows and Columns

The transpose of a matrix A, written A^T, flips the matrix over its main diagonal. Rows become columns; columns become rows.

If A has shape [M, N], then A^T has shape [N, M].

Element-wise: (A^T)_{ij} = A_{ji} — the element at row i, column j of the transpose equals the element at row j, column i of the original.

Transpose by Hand

Matrix A (shape [2, 3]):

A = [ 1  2  3 ]
    [ 4  5  6 ]

Transpose A^T (shape [3, 2]):

A^T = [ 1  4 ]
      [ 2  5 ]
      [ 3  6 ]

Row 0 of A becomes column 0 of A^T. Row 1 of A becomes column 1 of A^T.

Double Transpose Returns Original

(A^T)^T = A — transposing twice restores the original matrix.

---

Why Attention Needs Transposition: Q @ K^T

This is the critical shape-matching step in attention.

Suppose:

• Sequence length T = 3
• Head dimension d_k = 4

After projecting input X [1, 3, 4]:

• Query matrix Q: shape [1, 3, 4] (we drop the batch dim for clarity: [3, 4])
• Key matrix K: shape [1, 3, 4] (similarly [3, 4])

We want to compute attention scores between every query and every key. That means a dot product between each row of Q and each row of K.

If we try Q @ K directly:

• Q is [3, 4]
• K is [3, 4]
• Inner dimensions: 4 and 3 — MISMATCH! Multiplication is undefined.

Solution: Transpose K to K^T:

• K^T has shape [4, 3]
• Now Q @ K^T: [3, 4] @ [4, 3] → inner dim 4 matches ✓
• Result: [3, 3] — a square matrix of attention scores!

Each entry S[i][j] = Q[i] ⋅ K[j] = how much token i (query) attends to token j (key).

Visualizing the Shape Fix

       Q (3 × 4)       ×        K^T (4 × 3)        =        S (3 × 3)
   [ q_0 ]                [ k_0  k_1  k_2 ]               [ s_00 s_01 s_02 ]
   [ q_1 ]        @       [ ...              ]       =    [ s_10 s_11 s_12 ]
   [ q_2 ]                [ ...              ]               [ s_20 s_21 s_22 ]

Each q_i is a row vector of length 4.
Each k_j is a column vector of length 4 (row j of K, transposed).
s_ij = q_i ⋅ k_j = dot product of query i and key j.

---

Vector Norms: Measuring Length

The norm (or magnitude, length) of a vector u of dimension d is:

||u|| = sqrt( u_1^2 + u_2^2 + ... + u_d^2 )

This is the Euclidean distance from the origin to the point represented by u.

Example: Norm of [3, 4, 0]

||u|| = sqrt(3^2 + 4^2 + 0^2) = sqrt(9 + 16 + 0) = sqrt(25) = 5

The vector [3, 4, 0] has length 5 — it lies on a circle of radius 5 in the plane.

---

Vector Similarity: Dot Product and Cosine Similarity

The dot product measures both direction and length:

u ⋅ v = ||u|| ||v|| cos(θ)

Where θ is the angle between the vectors.

• If u and v point the same way (θ = 0): cos(θ) = 1, dot product = ||u|| ||v|| (maximum positive)
• If orthogonal (θ = 90°): cos(θ) = 0, dot product = 0
• If opposite (θ = 180°): cos(θ) = -1, dot product = -||u|| ||v|| (maximum negative)

Problem: Long vectors produce large dot products even if poorly aligned. A vector of length 100 pointing slightly toward another vector can outscore a perfectly aligned pair of length-1 vectors.

Solution: Cosine similarity normalizes by the lengths:

cos(θ) = (u ⋅ v) / (||u|| ||v||)

Range: [-1, 1]

• +1 = identical direction
• 0 = orthogonal
• -1 = opposite direction

Transformers use scaled dot-product attention: Q @ K^T / sqrt(d_k). Dividing by sqrt(d_k) has a similar normalizing effect, preventing gradients from vanishing or exploding when d_k is large.

---

Worked Exercises & Solutions

Exercise 1: Transpose a [2, 3] Matrix and Verify (A^T)^T = A

Problem: Given matrix A (shape [2, 3]), compute A^T, then compute (A^T)^T and confirm it equals A.

A = [ 2  -1  4 ]
    [ 0   3  1 ]

Solution:

Step 1: Compute A^T (shape [3, 2])

Row 0 of A → Column 0 of A^T: [2, 0] Row 1 of A → Column 1 of A^T: [-1, 3] Row 2 of A → Column 2 of A^T: [4, 1]

A^T = [ 2  0 ]
      [ -1  3 ]
      [ 4  1 ]

Step 2: Compute (A^T)^T (shape [2, 3])

Row 0 of A^T → Column 0: [2, -1, 4] Row 1 of A^T → Column 1: [0, 3, 1]

(A^T)^T = [ 2  -1  4 ]
          [ 0   3  1 ]

Result: (A^T)^T = A ✓

---

Exercise 2: Vector Similarity Calculation

Problem: Given u = [3, 4, 0] and v = [4, 3, 0], compute:

• ||u|| (norm of u)
• ||v|| (norm of v)
• u ⋅ v (dot product)
• Cosine similarity cos(θ) = (u ⋅ v) / (||u|| ||v||)

Solution:

1. Norm of u:

||u|| = sqrt(3^2 + 4^2 + 0^2) = sqrt(9 + 16) = sqrt(25) = 5

2. Norm of v:

||v|| = sqrt(4^2 + 3^2 + 0^2) = sqrt(16 + 9) = sqrt(25) = 5

3. Dot product u ⋅ v:

u ⋅ v = 3×4 + 4×3 + 0×0 = 12 + 12 + 0 = 24

4. Cosine similarity:

cos(θ) = 24 / (5 × 5) = 24 / 25 = 0.96

Interpretation: The vectors have a cosine similarity of 0.96 — they point in very similar directions (both lie in the first quadrant, close to the line y=x). The raw dot product 24 is large partly because both vectors have length 5. Cosine similarity removes the length factor, revealing the pure directional alignment.

Note: All numbers in this exercise are exact (no rounding). The cosine 0.96 is 24/25 exactly.

---

Exercise 3: Shape Trace of Q and K Projections to Attention Scores

Problem: Trace tensor shapes through Query and Key projections to the attention score matrix.

Given:

• Input X: shape [1, 3, 4] (batch=1, seq_len=3, d_model=4)
• Query projection weight W_q: shape [3, 4] (d_head=3)
• Key projection weight W_k: shape [3, 4] (d_head=3)

Show the shapes of:

1. Q = X @ W_q^T
2. K = X @ W_k^T
3. K^T (transpose of K, swapping last two dims)
4. S = Q @ K^T (attention score matrix)

Solution:

1. Q shape:

X:       [1, 3, 4]
W_q^T:   [4, 3]   (transpose of [3, 4])
Q:       [1, 3, 3]   (inner dim 4 matches, output [1, 3, 3])

2. K shape:

X:       [1, 3, 4]
W_k^T:   [4, 3]
K:       [1, 3, 3]

3. K^T shape: Transpose swaps the last two dimensions (sequence and head_dim):

K:       [1, 3, 3]
K^T:     [1, 3, 3]   (square matrix, so shape unchanged)

4. S = Q @ K^T shape:

Q:       [1, 3, 3]
K^T:     [1, 3, 3]
S:       [1, 3, 3]   (batched matmul: [3,3] @ [3,3] = [3,3] per batch)

Result: The attention score matrix S has shape [1, 3, 3] — one 3×3 score matrix for the single batch, where S[0, i, j] = how much token i attends to token j.

Key observation: Because d_head = 3 and seq_len = 3, the score matrix is square (3×3). In general, S has shape [B, T, T] — always square in the sequence dimensions.

---

Summary Reference Table

Concept	Formula / Shape Rule	Purpose
Linear projection	[..., d_in] @ [d_out, d_in]^T = [..., d_out]	Map vectors to new feature space (Q, K, V)
Transpose	A [M, N] → A^T [N, M], (A^T)_{ij} = A_{ji}	Swap rows and columns
Double transpose	(A^T)^T = A	Identity operation
Attention score shape	Q [T, d_k] @ K^T [d_k, T] = S [T, T]	Every token attends to every token
Vector norm	‖u‖ = sqrt(Σ u_i^2)	Geometric length
Dot product	u ⋅ v = Σ u_i v_i = ‖u‖ ‖v‖ cos(θ)	Alignment × length
Cosine similarity	cos(θ) = (u ⋅ v) / (‖u‖ ‖v‖)	Pure direction alignment, range [-1, 1]
Scaled dot product	Q @ K^T / sqrt(d_k)	Stabilize attention gradients

---

Primary Source Citations

• Attention projections and QK^T: Vaswani et al. (2017), Attention Is All You Need, NeurIPS 2017 (arXiv:1706.03762) — Sections 3.1 and 3.2.1: Linear projections Q = X W_Q, K = X W_K, V = X W_V and scaled dot-product attention softmax(Q K^T / sqrt(d_k)) V.
• Norms, angles, cosine similarity: Goodfellow, Bengio, & Courville (2016), Deep Learning, MIT Press, Chapter 2 (Linear Algebra for Deep Learning) — Sections 2.5 (Norms) and 2.6 (Special Kinds of Matrices and Vectors).

---

What Comes Next

• Next lesson: Softmax, Probability, and Sampling by Hand — Turn the raw attention scores into a probability distribution and sample the next token.
• Previous lesson: Vectors, Matrices, and Tensor Shapes
• Track overview: Start Here: LLM Foundations