Normalization, Computation Cost, and Number Formats

In Softmax, Probability, and Sampling by Hand, you learned how raw logits become valid probabilities and how the max-subtraction trick keeps the computation numerically safe. That lesson concluded with the two places softmax appears in every transformer: inside attention weights and at the output head.

This lesson steps back from probabilities and asks three broader questions that govern every inference system:

1. Why normalize? How do deep transformer stacks avoid activations that explode to infinity or vanish to zero?
2. Where does compute go? How does the cost of attention grow compared to feed-forward layers as sequences get longer?
3. How much memory? How do number formats — FP32, FP16, BF16, FP8, INT8, INT4 — determine whether a model fits on your hardware?

---

Why Normalization Matters

A decoder-only transformer repeats the same pattern across many layers (often 32, 48, or even 80+ layers). Each layer takes an input vector, transforms it through weight matrices, adds it back to a residual stream (the running sum of all previous layer outputs), and passes the result to the next layer.

Without any normalization, two failure modes appear:

• Exploding activations: Repeated additions and matrix multiplications can cause values to grow without bound. Eventually you hit floating-point overflow, producing inf or NaN (Not a Number). Inference crashes.
• Vanishing activations: Values can shrink toward zero, destroying the signal the model learned. Gradients also vanish during training, preventing learning entirely.

Normalization solves both by forcing the activations into a stable range at every layer — typically a unit-scale distribution centered at zero. The model learns a per-feature gain (scale) parameter to recover expressivity.

---

LayerNorm vs. RMSNorm

LayerNorm (Ba, Kiros, & Hinton, 2016)

LayerNorm computes the mean and variance of each feature vector, shifts by subtracting the mean, divides by the standard deviation, then applies learned scale (γ, “gamma”) and shift (β, “beta”) parameters.

For a vector x of length d:

μ = (1/d) * Σ x_i
σ² = (1/d) * Σ (x_i - μ)²
x_norm_i = (x_i - μ) / sqrt(σ² + ε) * γ_i + β_i

RMSNorm (Zhang & Sennrich, 2019)

Zhang & Sennrich demonstrated that the mean-centering step is unnecessary. Scaling by the Root Mean Square (RMS) alone provides identical training stability and performance, while eliminating the mean calculation pass and saving 7% to 64% execution time on GPU kernels.

For a vector x of length d with learned gain γ (no shift β):

RMS(x) = sqrt( (1/d) * Σ x_i² + ε )
x_norm_i = (x_i / RMS(x)) * γ_i

Where ε (epsilon) is a tiny constant like 1e-5 or 1e-6 to prevent division by zero.

Worked Example: RMSNorm on [2.0, -2.0, 4.0, -4.0]

Let’s compute RMSNorm step by step with d = 4, γ = [1, 1, 1, 1], and ε = 0 for clarity.

Input vector: x = [2.0, -2.0, 4.0, -4.0]

Step 1: Sum of squares

2.0²   = 4.0
(-2.0)² = 4.0
4.0²   = 16.0
(-4.0)² = 16.0
Sum = 4 + 4 + 16 + 16 = 40.0

Step 2: Mean square

Mean square = 40.0 / 4 = 10.0

Step 3: RMS

RMS(x) = sqrt(10.0) ≈ 3.1623

Step 4: Normalize (divide by RMS, then scale by γ=1)

x_norm = [2.0/3.1623, -2.0/3.1623, 4.0/3.1623, -4.0/3.1623]
       ≈ [0.6325, -0.6325, 1.2649, -1.2649]

Step 5: Verify — the RMS of the normalized vector is exactly 1.0!

0.6325² ≈ 0.4000
(-0.6325)² ≈ 0.4000
1.2649² ≈ 1.6000
(-1.2649)² ≈ 1.6000
Sum = 0.4 + 0.4 + 1.6 + 1.6 = 4.0
Mean square = 4.0 / 4 = 1.0
RMS = sqrt(1.0) = 1.0  ✓

Key insight: RMSNorm guarantees every normalized vector has RMS = 1.0, so all feature vectors live on the same scale. The learned γ lets each feature choose its own effective magnitude.

---

Big-O Complexity: Attention vs. Feed-Forward Layers

Two computational kernels dominate transformer inference. Understanding their scaling behavior is essential for capacity planning and bottleneck analysis.

Variables

Symbol	Meaning
T	Sequence length (number of tokens in the context)
d	Model hidden dimension (e.g., 4096 for Llama-2-7B)

Feed-Forward Layers (Linear Projections & FFN)

Every linear projection (W_q, W_k, W_v, W_o, and the two FFN matrices) multiplies a [T, d] input by a [d, d] weight matrix.

FLOPs for one projection: 2 * T * d * d = 2 * T * d²

Scaling: O(T * d²)

• Scales linearly with sequence length T
• Scales quadratically with hidden dimension d

Self-Attention (Q @ Kᵀ and Attn @ V)

Self-attention performs two matrix multiplications involving the sequence length twice:

1. Q @ Kᵀ: [T, d] × [d, T] → [T, T] — 2 * T² * d FLOPs
2. Attn @ V: [T, T] × [T, d] → [T, d] — 2 * T² * d FLOPs

Total attention FLOPs: 4 * T² * d

Scaling: O(T² * d)

• Scales quadratically with sequence length T!
• Scales linearly with hidden dimension d

The Crossover Point

For short sequences (e.g., T = 128, d = 4096):

• T * d² = 128 * 4096² ≈ 2.15 × 10⁹ (FFN dominates)
• T² * d = 128² * 4096 ≈ 6.71 × 10⁷ (attention is small)

For long sequences (e.g., T = 32,768, d = 4096):

• T * d² = 32768 * 4096² ≈ 5.50 × 10¹¹
• T² * d = 32768² * 4096 ≈ 4.40 × 10¹² (attention dominates by 8×, because T / d = 32,768 / 4096 = 8)

Conclusion: As context length grows, attention becomes the overwhelming computational and memory bottleneck. This is why FlashAttention, PagedAttention, and other kernel optimizations target the T² term specifically.

---

Number Formats and Precision in Inference

When you download a model checkpoint, the weights are stored in some numeric format. The format determines three things:

1. Memory footprint — bytes per parameter
2. Dynamic range — the largest/smallest non-zero values representable (controlled by exponent bits)
3. Precision — how finely values can be distinguished (controlled by mantissa/fraction bits)

Format	Total Bits	Bytes	Sign	Exponent	Mantissa	Max Finite Value (approx.)	Use Case
FP32	32	4	1	8	23	3.4 × 10³⁸	Training baseline, high accuracy
FP16	16	2	1	5	10	6.5 × 10⁴	Older GPUs; overflows easily
BF16	16	2	1	8	7	3.4 × 10³⁸	De facto standard for LLMs (Llama, Mistral)
FP8 (E4M3)	8	1	1	4	3	240	Weights/activations on Hopper+
FP8 (E5M2)	8	1	1	5	2	5.7 × 10⁴	Wider dynamic range
INT8	8	1	1*	—	7*	127 (signed)	Quantized weights, symmetric
INT4	4	0.5	1*	—	3*	7 (signed)	Extreme compression (AWQ, GPTQ)

_INT formats use a shared scale factor s per tensor or channel: x ≈ s _ qwhereq is the integer.*

Format Details

FP32 (Single Precision) — The IEEE 754 standard 32-bit float. Highest accuracy but 4× the memory of FP8/INT8. Used for training and as the reference baseline.

FP16 (Half Precision) — 16 bits with only 5 exponent bits. Maximum finite value ~65,504. During training, gradients or activations easily overflow to infinity. Largely replaced by BF16 for LLM work.

BF16 (Brain Floating Point) — 16 bits with the same 8 exponent bits as FP32 (same dynamic range), but only 7 mantissa bits (lower precision). Prevents overflow while halving memory vs FP32. The de facto standard for modern LLM inference (Llama 2/3, Mistral, etc.).

FP8 — 8 bits = 1 byte. Two variants on Hopper (H100) and Ada Lovelace (RTX 40-series):

• E4M3 (4 exponent, 3 mantissa): Higher precision, used for weights and activations
• E5M2 (5 exponent, 2 mantissa): Wider range, used for gradients or master weights

INT8 (8-bit Integer) — Fixed-point quantization. Weights are mapped to integers -128..127 with a per-tensor or per-channel scale factor x ≈ s * q. Supported on all modern GPUs via Tensor Cores.

INT4 (4-bit Integer) — 4 bits = 0.5 bytes. Weights quantized to 16 discrete values. Methods like AWQ (Activation-aware Weight Quantization) and GPTQ calibrate scales to preserve quality. Halves memory footprint again vs INT8.

---

VRAM Footprint Calculation by Hand

Formula:

Weight Memory (GB) = (Parameters × Bytes per parameter) / 10⁹

7B Model Example (7 × 10⁹ parameters)

Precision	Bytes/Param	Calculation	VRAM
FP32	4	7 × 4	28 GB
FP16 / BF16	2	7 × 2	14 GB
FP8 / INT8	1	7 × 1	7 GB
INT4	0.5	7 × 0.5	3.5 GB

Why quantization is essential:

• An RTX 4090 has 24 GB VRAM — cannot fit 7B in FP32 (28 GB), fits comfortably in BF16 (14 GB) or INT4 (3.5 GB).
• A consumer laptop with 16 GB unified memory — cannot run 7B in FP32 or BF16, but runs INT4 smoothly.

70B Model Example (70 × 10⁹ parameters)

Precision	Bytes/Param	Calculation	VRAM	Typical Hardware
BF16	2	70 × 2	140 GB	2× A100/H100 80 GB
INT8	1	70 × 1	70 GB	1× A100/H100 80 GB
INT4	0.5	70 × 0.5	35 GB	1× RTX 6000 Ada (48 GB)

Note: These are weight-only numbers. The KV cache adds additional memory proportional to 2 * layers * num_kv_heads * head_dim * context * bytes_per_param. See the KV Cache lessons for that calculation.

---

Hands-On Exercises & Checked Solutions

Exercise 1: Hand-Computing RMSNorm

Problem: Compute RMSNorm for x = [3.0, 1.0, -1.0, -3.0] with d = 4, γ = [1, 1, 1, 1], ε = 0.

Solution:

Step 1: Sum of squares

3.0²   = 9.0
1.0²   = 1.0
(-1.0)² = 1.0
(-3.0)² = 9.0
Sum = 9 + 1 + 1 + 9 = 20.0

Step 2: Mean square

Mean square = 20.0 / 4 = 5.0

Step 3: RMS

RMS(x) = sqrt(5.0) ≈ 2.2361

Step 4: Normalize

x_norm = [3.0/2.2361, 1.0/2.2361, -1.0/2.2361, -3.0/2.2361]
       ≈ [1.3416, 0.4472, -0.4472, -1.3416]

Step 5: Verify RMS of normalized vector = 1.0

1.3416² ≈ 1.8000
0.4472² ≈ 0.2000
(-0.4472)² ≈ 0.2000
(-1.3416)² ≈ 1.8000
Sum = 1.8 + 0.2 + 0.2 + 1.8 = 4.0
Mean square = 4.0 / 4 = 1.0
RMS = sqrt(1.0) = 1.0  ✓

---

Exercise 2: Scaling Growth — Attention vs. Feed-Forward

Problem: Compare operational growth as T increases from 512 to 8192 (a 16× increase) with fixed d = 2048.

Solution:

Feed-Forward cost (proportional to T * d²):

T=512:  512 × 2048² = 512 × 4,194,304 = 2.147 × 10⁹
T=8192: 8192 × 2048² = 8192 × 4,194,304 = 3.436 × 10¹⁰
Growth = 3.436e10 / 2.147e9 = 16.0×   (exactly 16×, linear in T)

Attention cost (proportional to T² * d):

T=512:  512² × 2048 = 262,144 × 2048 = 5.369 × 10⁸
T=8192: 8192² × 2048 = 67,108,864 × 2048 = 1.374 × 10¹¹
Growth = 1.374e11 / 5.369e8 = 256×   (16² = 256, quadratic in T!)

Conclusion: A 16× increase in sequence length causes only 16× more FFN work, but 256× more attention work. Long context shifts the bottleneck decisively to attention.

---

Exercise 3: Hardware Memory Budgeting for a 70B Model

Problem: Compute exact weight VRAM for a 70B parameter model in three precisions.

Solution:

1. BF16 (2 bytes/param)

VRAM = 70 × 10⁹ × 2 / 10⁹ = 140 GB

Requires 2× 80 GB GPUs (e.g., 2× A100-80GB or 2× H100-80GB).

2. INT8 (1 byte/param)

VRAM = 70 × 10⁹ × 1 / 10⁹ = 70 GB

Fits on a single 80 GB GPU (A100-80GB or H100-80GB).

3. INT4 (0.5 bytes/param)

VRAM = 70 × 10⁹ × 0.5 / 10⁹ = 35 GB

Fits on a single 48 GB workstation GPU (e.g., RTX 6000 Ada Generation).

---

Summary Reference Table

Topic	Key Formula / Rule	Practical Implication
LayerNorm	x_norm = (x - μ) / σ * γ + β	Mean + variance pass; 2× reads
RMSNorm	x_norm = x / RMS(x) * γ	No mean pass; 7–64% faster kernels
RMS(x)	sqrt( (1/d) * Σ x_i² + ε )	Guarantees unit RMS after norm
FFN Complexity	O(T * d²)	Linear in sequence length
Attention Complexity	O(T² * d)	Quadratic in sequence length
Crossover	T ≈ d	When T > d, attention dominates
FP32	4 bytes/param	28 GB for 7B; training reference
BF16	2 bytes/param	14 GB for 7B; standard for LLMs
FP8 / INT8	1 byte/param	7 GB for 7B; single consumer GPU
INT4	0.5 bytes/param	3.5 GB for 7B; laptop-friendly

---

Primary Source Citations

• RMSNorm: Zhang & Sennrich (2019), Root Mean Square Layer Normalization, NeurIPS 2019 — arXiv:1910.07467
• LayerNorm: Ba, Kiros, & Hinton (2016), Layer Normalization, arXiv — arXiv:1607.06450
• Attention complexity: Vaswani et al. (2017), Attention Is All You Need, Section 3.2 and 3.3 — arXiv:1706.03762
• INT8 quantization: Dettmers et al. (2022), LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale, NeurIPS 2022 — arXiv:2208.07339
• AWQ (INT4): Lin et al. (2023), AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration, MLSys 2024 — arXiv:2306.00978

---

What Comes Next

• Next lesson: Token Embeddings and Position in a Decoder — Follow token vectors into the model, then show how positional information, including RoPE, changes attention inputs.
• Previous lesson: Softmax, Probability, and Sampling by Hand
• Track overview: Start Here: LLM Foundations