Vectors, Matrices, and Tensor Shapes

In From Logits to the Next Token, you saw the transformer’s final hidden state (a vector of size 4 in our toy model) multiplied by the output projection matrix of shape [7, 4] to produce 7 logits — one per vocabulary token. That multiplication is matrix multiplication, and the shapes [1, 3, 4] (input tensor) and [7, 4] (weight matrix) are tensor shapes.

This lesson builds the linear-algebra toolkit used in every transformer calculation. You will learn to distinguish scalars, vectors, matrices, and higher-dimensional tensors; calculate dot products and matrix products by hand; and track tensor shapes through the embedding and projection stages of a transformer.

---

Scalars, Vectors, Matrices, and Tensors

Scalar (0D Tensor)

A scalar is a single standalone number. It has no dimensions — its shape is written as () or [].

Examples in LLM inference:

• Temperature T = 0.7
• Scale factor sqrt(d_k) = 2.0 (for d_k = 4)
• Learning rate 0.001
• A single logit value 2.1

Scalar:  2.1
Shape:   ()

---

Vector (1D Tensor)

A vector is an ordered 1D array of numbers. Its length is its dimension d. In code, it is a flat list; on paper, we write it as a column or row.

Examples:

• Token embedding for a single token: [0.42, -0.15, 0.88, 0.05] → shape [4]
• Logits vector for a vocabulary of size 7: [ -3.2, 0.8, 1.2, 0.5, -0.4, 2.1, 0.9 ] → shape [7]
• A single row or column of a matrix

Vector:  [1.0, -0.5, 0.2, 1.8]
Shape:   [4]
Dimension: 4

Notation: We denote vectors with lowercase bold letters: v, u, x. The i-th element is v_i (1-indexed) or v[i] (0-indexed in code).

---

Matrix (2D Tensor)

A matrix is a rectangular 2D table of numbers with rows and columns. Its shape is [rows, cols] = [M, N].

Examples:

• A batch of token embeddings: 3 tokens × 4 features → shape [3, 4]
• Output projection (unembedding) weight matrix: vocabulary size 7 × embedding dim 4 → shape [7, 4]
• Query, Key, Value projection matrices in attention

Matrix (3 rows, 4 cols):
[ 0.42  -0.15   0.88   0.05 ]
[ -0.73  0.61   0.12   0.94 ]
[ 0.25  -0.80   0.33  -0.18 ]
Shape: [3, 4]

Notation: We denote matrices with uppercase bold letters: A, B, W. The element at row i, column j is A_ij (1-indexed) or A[i][j] (0-indexed in code).

---

Tensor (3D and Higher)

A tensor generalizes scalars, vectors, and matrices to any number of dimensions. In transformer inference, the standard input tensor has 3 dimensions:

• Batch size (B): Number of independent sequences processed together
• Sequence length (T): Number of tokens in each sequence
• Embedding dimension (d): Number of features per token

Shape notation: [B, T, d] or [Batch, Sequence_Length, Embedding_Dim]

Our running example from Lesson 2:

Prompt: "The cat sat"
Token IDs: [1, 2, 3]
Batch size B = 1
Sequence length T = 3
Embedding dim d = 4

Input tensor shape: [1, 3, 4]

Visualizing the 3D tensor:

Batch 0 (only batch):
  Position 0 ("The"):  [ 0.42, -0.15,  0.88,  0.05]
  Position 1 ("cat"):  [-0.73,  0.61,  0.12,  0.94]
  Position 2 ("sat"):  [ 0.25, -0.80,  0.33, -0.18]

Total numbers stored: B × T × d = 1 × 3 × 4 = 12 scalars.

---

The Dot Product (Inner Product)

The dot product of two equal-length vectors u and v (both of dimension d) is a single scalar computed by multiplying matching elements and summing the results:

u ⋅ v = u_1 v_1 + u_2 v_2 + ... + u_d v_d

In code notation: sum(u[i] * v[i] for i in range(d))

Why the Dot Product Matters

• Measures alignment: Positive → vectors point roughly the same direction. Zero → orthogonal (unrelated). Negative → opposing directions.
• Core of attention: Scaled dot-product attention computes Q ⋅ K^T — dot products between query and key vectors.
• Probability scoring: The output projection computes dot products between the final hidden state and each row of the unembedding matrix.

Worked Example: Dot Product by Hand

Vectors:

u = [1, 2, -1, 3]
v = [2, 0, 4, 1]

Step-by-step:

Index i	u_i	v_i	u_i × v_i (product)
1	1	2	2
2	2	0	0
3	-1	4	-4
4	3	1	3

Sum of products: 2 + 0 + (-4) + 3 = 1

Result: u ⋅ v = 1 (a scalar)

Interpretation: The dot product is 1 (small positive). These vectors have some weak alignment — not strongly pointing the same way, not orthogonal, not opposing.

---

Matrix Multiplication: The Row-by-Column Rule

Matrix multiplication is the engine of neural networks. Every linear projection, attention score, and feed-forward layer is a matrix multiply.

The Golden Shape Rule (Inner Dimension Match)

To multiply matrix A of shape [M, K] by matrix B of shape [K, N]:

• The inner dimension K MUST be identical.
• The result C has shape [M, N].

      A (M × K)    ×    B (K × N)    =    C (M × N)
    [M, K]               [K, N]             [M, N]
       ↑                   ↑                  ↑
    outer dim           inner dim         outer dim
    (rows of A)         (cols of A =      (cols of B)
                         rows of B)

If the inner dimensions don’t match (e.g., [2, 3] × [4, 5]), the multiplication is undefined.

Computing Each Entry: Row i of A Dotted with Column j of B

The entry C[i][j] (row i, column j of the result) is the dot product of row i of A and column j of B:

C[i][j] = sum_over_k ( A[i][k] * B[k][j] )

This is why the inner dimension must match: row i of A has length K, column j of B has length K. The dot product requires equal lengths.

Worked Example: Matrix Multiply by Hand

Matrix A (shape [2, 3]):

[ 1  2  3 ]
[ 4  5  6 ]

Matrix B (shape [3, 2]):

[ 7  8 ]
[ 9  1 ]
[ 2  3 ]

Check shapes: A is [2, 3], B is [3, 2] → inner dim = 3 ✓ → Result C is [2, 2]

Compute C[0][0] (row 0 of A ⋅ col 0 of B):

A row 0: [1, 2, 3]
B col 0: [7, 9, 2]
Dot: 1×7 + 2×9 + 3×2 = 7 + 18 + 6 = 31

Compute C[0][1] (row 0 of A ⋅ col 1 of B):

A row 0: [1, 2, 3]
B col 1: [8, 1, 3]
Dot: 1×8 + 2×1 + 3×3 = 8 + 2 + 9 = 19

Compute C[1][0] (row 1 of A ⋅ col 0 of B):

A row 1: [4, 5, 6]
B col 0: [7, 9, 2]
Dot: 4×7 + 5×9 + 6×2 = 28 + 45 + 12 = 85

Compute C[1][1] (row 1 of A ⋅ col 1 of B):

A row 1: [4, 5, 6]
B col 1: [8, 1, 3]
Dot: 4×8 + 5×1 + 6×3 = 32 + 5 + 18 = 55

Result C (shape [2, 2]):

[ 31  19 ]
[ 85  55 ]

---

Tensor Shape Tracking in Transformer Stages

Let’s trace shapes through the embedding lookup and output projection — the bookends of a transformer forward pass.

Stage 1: Embedding Lookup

From Lesson 2:

• Vocabulary size V = 7
• Embedding dimension d = 4
• Embedding table E: shape [V, d] = [7, 4]
• Input token IDs: [1, 2, 3] (sequence length T = 3)
• Batch size B = 1

Lookup operation: For each token ID, retrieve row ID from E.

Resulting tensor X has shape [B, T, d] = [1, 3, 4]:

X[0, 0, :] = E[1, :]  = [ 0.42, -0.15,  0.88,  0.05]  ("The")
X[0, 1, :] = E[2, :]  = [-0.73,  0.61,  0.12,  0.94]  ("cat")
X[0, 2, :] = E[3, :]  = [ 0.25, -0.80,  0.33, -0.18]  ("sat")

Stage 2: Output Projection (Unembedding)

From Lesson 3:

• Final hidden state for last position: shape [d] = [4] (after all transformer layers)
• Output projection matrix W_out: shape [V, d] = [7, 4]
• Logits = W_out × h_last (matrix-vector multiply)

Shape check: W_out is [7, 4], h_last is [4] (treated as [4, 1] column vector).

• Inner dim = 4 ✓
• Result shape = [7, 1] → squeeze to [7] (one logit per vocab token)

This matches the logits table in Lesson 3: 7 numbers for 7 vocabulary tokens.

General Rule: Linear Layer Shape Tracking

A linear layer (matrix multiply with optional bias) transforms an input tensor:

Input:  [..., d_in]   (any leading batch/sequence dims, last dim = d_in)
Weight: [d_out, d_in]
Output: [..., d_out]

The last dimension of the input must match the second dimension of the weight matrix. All leading dimensions are batch dimensions and are preserved.

---

Counting FLOPs in Matrix Multiplication

FLOPs = Floating-Point Operations. In deep learning, we count both multiplications and additions.

Multiplying A [M, K] by B [K, N]:

• Output matrix C has M × N entries
• Each entry requires K multiplications and K - 1 additions
• Standard convention: count 1 multiply + 1 add = 2 FLOPs per inner iteration
• Total FLOPs = 2 × M × K × N

Example: Output Projection FLOPs

• W_out: [7, 4] (V=7, d=4)
• h_last: [4] (or [4, 1])
• M = 7, K = 4, N = 1
• FLOPs = 2 × 7 × 4 × 1 = 56 FLOPs

Example: Embedding Lookup (Not a MatMul)

Embedding lookup is an indexing/gather operation, not a matrix multiplication. It reads T rows of d elements each = T × d memory reads. No arithmetic FLOPs.

---

Worked Exercises & Solutions

Exercise 1: Dot Product of Two 4D Vectors

Problem: Compute the dot product of u = [1, 2, -1, 3] and v = [2, 0, 4, 1]. Show each intermediate product and the final sum.

Solution:

i	u_i	v_i	u_i × v_i
1	1	2	2
2	2	0	0
3	-1	4	-4
4	3	1	3

Sum: 2 + 0 + (-4) + 3 = 1

Answer: u ⋅ v = 1

---

Exercise 2: Matrix Multiplication by Hand

Problem: Multiply matrix A (shape [2, 3]) by matrix B (shape [3, 2]). Show all row-by-column dot products.

A =

[ 1  2  3 ]
[ 4  5  6 ]

B =

[ 7  8 ]
[ 9  1 ]
[ 2  3 ]

Solution:

Shape check: [2, 3] × [3, 2] → inner dim 3 matches ✓ → Result [2, 2]

C[0][0] = row 0 of A ⋅ col 0 of B = [1,2,3] ⋅ [7,9,2] = 1×7 + 2×9 + 3×2 = 7 + 18 + 6 = 31

C[0][1] = row 0 of A ⋅ col 1 of B = [1,2,3] ⋅ [8,1,3] = 1×8 + 2×1 + 3×3 = 8 + 2 + 9 = 19

C[1][0] = row 1 of A ⋅ col 0 of B = [4,5,6] ⋅ [7,9,2] = 4×7 + 5×9 + 6×2 = 28 + 45 + 12 = 85

C[1][1] = row 1 of A ⋅ col 1 of B = [4,5,6] ⋅ [8,1,3] = 4×8 + 5×1 + 6×3 = 32 + 5 + 18 = 55

Result C:

[ 31  19 ]
[ 85  55 ]

---

Exercise 3: Transformer Shape Tracking and FLOPs

Problem: A mini-transformer processes a prompt of 4 tokens with embedding dimension 8. The input tensor has shape [1, 4, 8]. The final hidden state for the last position (shape [8]) is multiplied by an output projection weight matrix of shape [16, 8] (vocabulary size 16, embedding dim 8).

1. What is the shape of the output logits vector?
2. How many FLOPs does this output projection require?
3. If the batch size were increased to 32 (shape [32, 4, 8]), and we projected all 4 positions per sequence (not just the last), how many FLOPs for the full projection?

Solution:

1. Output logits shape:

• Weight W: [16, 8]
• Hidden state h: [8] (treated as [8, 1])
• Inner dim 8 matches ✓
• Result: [16, 1] → squeeze to [16]
• Answer: [16] (16 logits, one per vocabulary token)

2. FLOPs for single-position projection:

• M = 16, K = 8, N = 1
• FLOPs = 2 × 16 × 8 × 1 = 256
• Answer: 256 FLOPs

3. FLOPs for full batch × sequence projection:

• Input: [32, 4, 8] → reshape to [128, 8] (32 × 4 = 128 tokens)
• Weight: [16, 8]
• Multiply [128, 8] × [8, 16] (note: we transpose weight for X W^T convention, but FLOPs same)
• M = 128, K = 8, N = 16
• FLOPs = 2 × 128 × 8 × 16 = 32,768
• Answer: 32,768 FLOPs

---

Summary Reference Table

Concept	Shape / Formula	Key Rule
Scalar	() or []	Single number
Vector	[d]	Ordered list of d numbers
Matrix	[M, N]	M rows, N columns
Tensor (3D)	[B, T, d]	Batch, Sequence, Embedding
Dot product	u ⋅ v = Σ u_i v_i	Vectors must have same length d; result is scalar
Matrix multiply	[M, K] × [K, N] = [M, N]	Inner dim K must match; C[i][j] = row_i(A) ⋅ col_j(B)
Linear layer	[..., d_in] × [d_out, d_in] = [..., d_out]	Last input dim = weight second dim
FLOPs (matmul)	2 × M × K × N	Counts multiply + add per inner iteration

---

Primary Source Citations

• Linear algebra for deep learning: Goodfellow, Bengio, & Courville (2016), Deep Learning, MIT Press, Chapter 2 (Linear Algebra for Deep Learning).
• Transformer projections: Vaswani et al. (2017), Attention Is All You Need, NeurIPS 2017 (arXiv:1706.03762) — Input embeddings V × d, output projection V × d (weight tying), Q/K/V projections.

---

What Comes Next

• Next lesson: Linear Projections, Transposes, and Similarity — Follow the shapes of a small linear projection and transpose, then use a dot product to compare two vectors.
• Previous lesson: From Logits to the Next Token
• Track overview: Start Here: LLM Foundations