Calculate KV Cache Memory by Hand

In the previous lesson, you learned why a decoder-only transformer caches Keys and Values: causal attention makes past K and V vectors invariant to future tokens, so we compute them once and reuse them forever. You also saw their tensor shapes at a single layer:

K_layer ∈ [B, H_kv, T_seq, d_head]
V_layer ∈ [B, H_kv, T_seq, d_head]

Now we go from shapes to bytes. This lesson derives the complete memory formula from a single scalar number, explains every factor’s physical origin, compares the three major attention head architectures, and gives you hands-on exercises to calculate real-world memory footprints and concurrency limits.

---

The First-Principles Derivation: Building Up from One Scalar

We start with the smallest possible unit — one scalar number in memory — and multiply by each architectural factor that expands the cache.

Step 1: One token, one layer, one head

At a single layer ℓ, a single token position t, and a single key/value head h, the cache stores:

• One Key vector of length d_head → d_head scalars
• One Value vector of length d_head → d_head scalars

Total scalars per token per layer per head = 2 × d_head

Why 2? Because every token produces exactly one Key and one Value. The factor of 2 is not “magic” — it directly counts the two vectors.

Step 2: All key/value heads in the layer

Modern attention architectures use multiple key/value heads per layer. Let H_kv be the number of key/value heads at this layer.

Total scalars per token per layer = 2 × d_head × H_kv

Step 3: All layers in the model

The transformer stacks L layers vertically. Each layer maintains its own independent KV cache.

Total scalars per token = 2 × d_head × H_kv × L

Step 4: Full sequence length

During generation, the sequence grows from the prompt length to the final length. At any moment, the cache holds Keys and Values for all tokens seen so far. Let S be the current sequence length (prompt + generated tokens).

Total scalars for one request = 2 × d_head × H_kv × L × S

Step 5: Batched serving

A production server processes B independent requests concurrently. Each request has its own sequence, its own cache, and they do not share cache entries.

Total scalars for the batch = 2 × d_head × H_kv × L × S × B

Step 6: Precision — bytes per scalar

Every scalar element occupies b bytes in memory depending on the numeric format:

Format	Bytes per element (b)	Common use case
FP32	4	Training, reference
FP16 / BF16	2	Standard inference (A100, H100, RTX 30/40 series)
FP8 (E4M3 / E5M2)	1	H100 Tensor Cores, quantization-aware serving
INT8	1	Quantized weights and cache
INT4	0.5 (4 bits)	Extreme compression, llama.cpp, EXL2

The Complete Formula

KV_Cache_Bytes = 2 × B × S × L × H_kv × d_head × b

Every factor has a physical origin:

Symbol	Name	Physical meaning
2	Key + Value	Two vectors produced per token per head
B	Batch size	Concurrent independent requests
S	Sequence length	Prompt tokens + generated tokens so far
L	Layer count	Stacked decoder blocks, each with independent cache
H_kv	Key/Value head count	Parallel attention heads producing K/V
d_head	Head dimension	Width of each K/V vector (d_model / H_q)
b	Bytes per element	Numeric precision (FP16=2, FP8=1, INT4=0.5)

---

Architectural Variants: MHA vs GQA vs MQA

The formula above uses H_kv — the number of key/value heads. This is where the three attention architectures differ dramatically.

Multi-Head Attention (MHA)

• Definition: Every query head has its own dedicated key head and value head.
• Relation: H_kv = H_q
• Examples: Llama 1, Llama 2 (7B: H_q = 32, H_kv = 32), GPT-3 175B
• Memory footprint: Highest. No sharing.

Grouped-Query Attention (GQA)

• Definition: Query heads are partitioned into G groups. All query heads in a group share one key head and one value head.
• Relation: H_kv = H_q / G (where G is the group size)
• Examples: Llama 3 8B (H_q = 32, H_kv = 8, group size 4), Llama 3 70B (H_q = 64, H_kv = 8, group size 8), Gemma 7B (H_q = 32, H_kv = 8)
• Memory footprint: G× reduction vs MHA. For Llama 3 8B: 4× smaller cache!

Multi-Query Attention (MQA)

• Definition: One single key head and one single value head shared by all query heads.
• Relation: H_kv = 1
• Examples: Falcon 7B (H_q = 71, H_kv = 1), Falcon 40B, PaLM
• Memory footprint: H_q× reduction vs MHA. Maximum sharing.

Visual Comparison

MHA:  Q1 Q2 Q3 Q4 ... Q32     → 32 KV heads
       │  │  │  │     │
       K1 K2 K3 K4 ... K32
       V1 V2 V3 V4 ... V32

GQA:  Q1 Q2 Q3 Q4 | Q5 Q6 Q7 Q8 | ... | Q29 Q30 Q31 Q32
       │  │  │  │     │  │  │  │          │  │  │  │
       K1        K2              ...         K8       (8 KV heads, group size 4)
       V1        V2              ...         V8

MQA:  Q1 Q2 Q3 Q4 ... Q71       → 1 KV head
       │  │  │  │     │
       K1
       V1

Key insight: The query projection still computes H_q separate query vectors — they just read from the same cached K/V. This preserves much of MHA’s representation power while slashing cache memory.

---

Units and Precision Conventions

Before we calculate, we must be precise about units. Confusion here leads to 7%–10% errors in capacity planning.

Gigabytes vs Gibibytes

Unit	Symbol	Definition	Bytes
Gigabyte	GB	10^9	1,000,000,000
Gibibyte	GiB	2^30	1,073,741,824

Conversion: 1 GiB = 1.073741824 GB

Throughout this lesson, we report both:

• GiB for GPU VRAM (hardware specsheets use GiB: 80 GiB A100, 24 GiB RTX 3090)
• GB for human-readable approximations

Precision Byte Sizes (Explicit)

Format	Bytes per element (b)	Bits per element
FP32	4	32
FP16 / BF16	2	16
FP8 (E4M3/E5M2)	1	8
INT8	1	8
INT4	0.5	4

Note on INT4: 0.5 bytes means two INT4 values pack into one byte. In code this is a bit-packed array; in formulas we treat it as 0.5 bytes/element for clean scaling.

---

The Capacity Cliff: Static Weights vs Dynamic Cache

This is the most important mental model for serving engineers.

Model Weights (Static)

• Fixed at load time
• Does not change with batch size, sequence length, or precision of the cache
• Example: Llama 3 8B at BF16 ≈ 16 GiB (8 billion params × 2 bytes)

KV Cache (Dynamic)

• Grows linearly with B (batch size)
• Grows linearly with S (sequence length)
• Formula: 2 × B × S × L × H_kv × d_head × b

Total Memory at Runtime

Total_VRAM = Weights + KV_Cache(B, S) + Activations + System_Headroom

The Cliff

Scenario	Weights (GiB)	KV Cache (GiB)	Total (GiB)
B=1, S=4K (Llama 3 8B, BF16)	16	0.5	16.5
B=16, S=4K	16	8	24
B=64, S=4K	16	32	48
B=1, S=128K	16	16	32
B=16, S=128K	16	256	272 ⚠️

At high batch or long context, KV cache dwarfs the model weights and becomes the primary cause of GPU OOM errors.

This is why cache management (paging, quantization, offloading, eviction) dominates serving infrastructure design — covered in the next lessons.

---

Hands-On Exercise 1: Llama 2 7B (MHA) vs Llama 3 8B (GQA) at 4K Context

Goal: Calculate exact KV cache memory for two models at identical settings and measure the GQA reduction.

Given parameters (both models):

• L = 32 layers
• d_head = 128
• b = 2 bytes (BF16)
• S = 4096 tokens (4K context)
• B = 1 request

Model A: Llama 2 7B (MHA)

• H_kv = 32 (MHA: H_kv = H_q = 32)

Step-by-step calculation:

Elements = 2 × B × S × L × H_kv × d_head
         = 2 × 1 × 4096 × 32 × 32 × 128
         = 2 × 4096 × 32 × 4096
         = 2 × 4096 × 131072
         = 2 × 536,870,912
         = 1,073,741,824 elements

Bytes = Elements × b
      = 1,073,741,824 × 2
      = 2,147,483,648 bytes

GiB = Bytes / 2^30
    = 2,147,483,648 / 1,073,741,824
    = 2.0 GiB exactly

GB = Bytes / 10^9
   = 2,147,483,648 / 1,000,000,000
   ≈ 2.15 GB

Result: 2.0 GiB (≈ 2.15 GB) per active request

Model B: Llama 3 8B (GQA, group size 4)

• H_kv = 8 (GQA: H_kv = H_q / 4 = 32 / 4 = 8)

Step-by-step calculation:

Elements = 2 × 1 × 4096 × 32 × 8 × 128
         = 2 × 4096 × 32 × 1024
         = 2 × 4096 × 32768
         = 2 × 134,217,728
         = 268,435,456 elements

Bytes = 268,435,456 × 2
      = 536,870,912 bytes

GiB = 536,870,912 / 1,073,741,824
    = 0.5 GiB exactly

MiB = 0.5 × 1024
    = 512 MiB

GB = 536,870,912 / 10^9
   ≈ 536.9 MB

Result: 0.5 GiB (512 MiB, ≈ 536.9 MB) per active request

Comparison

Model	Architecture	H_kv	Cache per request	Reduction vs MHA
Llama 2 7B	MHA	32	2.0 GiB	1× (baseline)
Llama 3 8B	GQA (G=4)	8	0.5 GiB	4× smaller

Memory saved per request: 2.0 - 0.5 = 1.5 GiB

Takeaway: GQA with group size 4 gives you a 4× reduction in KV cache memory with minimal quality loss. This is why every major model released since mid-2023 (Llama 3, Gemma, Qwen 2, Nemotron) uses GQA.

---

Hands-On Exercise 2: Context Length and Batch Size Scaling Matrix

Goal: Build a 9-cell matrix showing how KV cache explodes with context and batch.

Model: Llama 3 8B

• L = 32, H_kv = 8, d_head = 128, b = 2 (BF16)

Simplified formula for this model:

Bytes = 2 × B × S × 32 × 8 × 128 × 2
      = 131,072 × B × S bytes
      = 128 KiB × B × S

Batch (B) \ Context (S)	4K (4,096)	32K (32,768)	128K (131,072)
B = 1	0.50 GiB	4.00 GiB	16.00 GiB
B = 16	8.00 GiB	64.00 GiB	256.00 GiB
B = 64	32.00 GiB	256.00 GiB	1024.00 GiB (1.0 TiB)

Verification of corner cells:

• B=1, S=4096: 131,072 × 1 × 4,096 = 536,870,912 bytes = 0.5 GiB ✓
• B=64, S=131,072: 131,072 × 64 × 131,072 = 1,099,511,627,776 bytes = 1024 GiB = 1.0 TiB ✓

Key Takeaways

1. At 128K context, a single request needs 16 GiB — that’s an entire A100 80GB GPU just for cache of one sequence!
2. At 128K context with batch 16, you need 256 GiB — requiring a 4×80GB GPU cluster just to hold the KV cache!
3. Batch size and context length multiply each other — the matrix is the outer product of B and S scaling.
4. This is why long-context serving requires: paged attention, cache quantization, CPU offloading, or disaggregated prefill/decode.

---

Hands-On Exercise 3: Maximizing Serving Concurrency Under a Fixed VRAM Budget

Goal: Calculate how many concurrent requests (B_max) fit on an A100 80 GB GPU under different cache precisions.

System Budget (A100 80 GB = 80 GiB)

Component	Allocation
Model weights (Llama 3 8B, BF16)	16 GiB
Activations (prefill + decode peak)	4 GiB
System headroom (OS, driver, fragmentation)	4 GiB
Remaining for KV cache	56 GiB

In bytes: 56 × 2^30 = 60,129,542,144 bytes

Workload Parameters

• Context length: S = 8192 (8K tokens)
• Model: Llama 3 8B (L = 32, H_kv = 8, d_head = 128)

Total elements per request across all layers:

Elements_per_request = 2 × S × L × H_kv × d_head
                     = 2 × 8192 × 32 × 8 × 128
                     = 536,870,912 elements

Memory per request by precision:

Precision	b (bytes)	Bytes per request	GiB per request
BF16	2	1,073,741,824	1.0 GiB
FP8	1	536,870,912	0.5 GiB
INT4	0.5	268,435,456	0.25 GiB

Case 1: BF16 Cache (b = 2)

Bytes per request = 536,870,912 × 2 = 1,073,741,824 bytes = 1.0 GiB
B_max = VRAM_for_cache / Bytes_per_request
      = 60,129,542,144 / 1,073,741,824
      = 56.0

Result: 56 concurrent requests

Case 2: FP8 Cache (b = 1)

Bytes per request = 1,073,741,824 / 2 = 536,870,912 bytes = 0.5 GiB
B_max = 56 GiB / 0.5 GiB = 112

Result: 112 concurrent requests (2× boost!)

Case 3: INT4 Cache (b = 0.5)

Bytes per request = 536,870,912 / 2 = 268,435,456 bytes = 0.25 GiB
B_max = 56 GiB / 0.25 GiB = 224

Result: 224 concurrent requests (4× boost!)

Summary Table

Cache Precision	Bytes/Request	GiB/Request	Max Concurrency (B_max)	Throughput Gain vs BF16
BF16	1,073,741,824	1.0 GiB	56	1× (baseline)
FP8	536,870,912	0.5 GiB	112	2×
INT4	268,435,456	0.25 GiB	224	4×

Critical caveat: Quantizing the KV cache to FP8 or INT4 introduces quality degradation (perplexity increase, downstream task accuracy drop). The concurrency gain is real, but you must measure quality on your workload before deploying. Lesson 19 (kv-precision-and-offloading) covers this tradeoff in depth.

---

Summary Reference Table: Popular Open Models

Model	L	H_q	H_kv	d_head	Architecture	Cache MiB per 1K tokens (BF16)
Llama 2 7B	32	32	32	128	MHA	512 MiB
Llama 3 8B	32	32	8	128	GQA (G=4)	128 MiB
Llama 3 70B	80	64	8	128	GQA (G=8)	320 MiB
Gemma 7B	28	32	8	256	GQA (G=4)	224 MiB
Falcon 7B	32	71	1	64	MQA	8 MiB
Falcon 40B	60	128	1	128	MQA	30 MiB
Mistral 7B	32	32	8	128	GQA (G=4)	128 MiB
Qwen 2.5 7B	28	32	8	128	GQA (G=4)	112 MiB
Nemotron 3 Ultra	96	128	16	128	GQA (G=8)	768 MiB

Formula for the table column:

MiB_per_1K = (2 × 1024 × L × H_kv × d_head × 2) / 2^20
           = (4096 × L × H_kv × d_head) / 1,048,576

---

Primary Source Citations

• Vaswani et al. (2017), Attention Is All You Need — arXiv:1706.03762 (Transformer architecture, scaled dot-product attention)
• Shazeer (2019), Fast Transformer Decoding: One Write-Head is All You Need — arXiv:1911.02150 (Multi-query attention, KV cache mechanics)
• Ainslie et al. (2023), GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — arXiv:2305.13245 (Grouped-Query Attention)
• Touvron et al. (2023), Llama 2: Open Foundation and Fine-Tuned Chat Models — arXiv:2307.09288 (Llama 2 architecture, MHA)
• AI at Meta (2024), The Llama 3 Herd of Models — arXiv:2407.21783 (Llama 3 architecture, GQA)
• Pope et al. (2022), Efficiently Scaling Transformer Inference — arXiv:2211.05102 (Prefill/decode, KV cache memory analysis)
• Kwon et al. (2023), Efficient Memory Management for Large Language Model Serving with PagedAttention — arXiv:2309.06180 (PagedAttention, KV cache memory management)

---

What Comes Next

• Next lesson: Allocate the Cache: Blocks, Waste, and PagedAttention — Now that you know how many bytes the cache needs, learn how to allocate those bytes without fragmentation, waste, or guessing sequence lengths.
• Previous lesson: Why Cache Keys and Values?
• Track overview: Start Here: LLM Foundations