Use Less Memory: Quantization and Offloading

In the previous lesson, you learned three ways to manage KV cache size when requests share prefixes or memory runs out: prefix caching (reuse identical blocks), cache eviction (discard or swap whole requests), and sliding-window attention (cap the cache at a fixed window W tokens per layer).

Those techniques decide which tokens to keep. This lesson teaches how to store each kept token using fewer bytes (quantization) and where to place tokens when GPU memory is full (tiered offloading).

---

1. Two Paths to Memory Relief

When the KV cache does not fit in GPU VRAM, you have two fundamentally different levers:

Path	Name	What Changes	Analogy
1	In-Place Compression (Quantization)	Each number in the cache is stored with fewer bits (FP16 → FP8 → INT8 → INT4). The data stays on the GPU.	Putting your files in a zip archive — same folder, smaller size.
2	Tiered Storage (Offloading)	Active tokens stay in fast GPU HBM; older tokens move to larger but slower memory tiers (CPU DDR5 RAM → NVMe SSD).	Moving old boxes from your desk to a storage unit — same boxes, different room.

Key insight: These paths are independent and composable. You can quantize the cache to INT8 (2× smaller), then offload the INT8 blocks to CPU RAM. The combined effect multiplies: INT8 + CPU offload = 2× (quantization) × 10–50× (capacity) = 20–100× effective VRAM relief.

---

2. KV Cache Quantization Mechanics

2.1 Symmetric Quantization Formula

The most common scheme for KV cache is symmetric quantization (zero point = 0). For a tensor x (Key or Value vector):

s = max(|x|) / q_max
x_q = clip(round(x / s), -q_max, q_max)
x_hat = x_q * s

Where:

• x = original FP16/BF16 value (or vector of values)
• s = scale factor (FP32, computed per quantization group)
• q_max = maximum representable integer (depends on target format)
• x_q = quantized integer value (stored in cache)
• x_hat = dequantized value (reconstructed during attention)

Target Format	q_max	Bits per Element	Bytes per Element
Signed INT8	127	8	1
Signed INT4	7	4	0.5

Why symmetric? KV cache values are roughly zero-centered (Keys and Values have mean ≈ 0 after RMSNorm). A zero point adds storage and compute overhead without benefit.

2.2 Granularity Matters: Per-Token or Per-Head Scaling

Per-tensor scaling (one scale for the entire KV cache) fails for attention. Keys have massive outlier channels caused by attention sinks (the first few tokens that absorb disproportionate attention mass). A single global scale would be dominated by these outliers, wasting precision on the remaining 99% of values.

Required granularity for KV cache:

Granularity	Scope	Use Case
Per-token	One scale per token position (applies to all heads/layers for that token)	Captures token-level dynamic range variation; standard for KV cache
Per-head	One scale per attention head per token	Finer control; more scale metadata
Per-channel (group)	One scale per group of channels (e.g., 128 channels) within a head	Used for weight quantization; overkill for KV cache

Practical standard: vLLM, SGLang, and TensorRT-LLM use per-token symmetric INT8/FP8 for KV cache. Each token carries its own scale factor (2 bytes FP16 or 4 bytes FP32) alongside the quantized values.

2.3 FP8 Formats: E4M3 vs E5M2

FP8 (8-bit floating point) has two standard layouts:

Format	Exponent Bits	Mantissa Bits	Dynamic Range	Precision	Best For
E4M3	4	3	±240 (max)	~3.3 bits	Activations, KV cache (higher precision, narrower range)
E5M2	5	2	±57344 (max)	~2.3 bits	Weights, gradients (wider range)

E4M3 is the standard for KV cache on NVIDIA Hopper (H100) and Ada (RTX 40-series) because:

• KV cache values stay in a narrow range (no extreme outliers after per-token scaling)
• Higher mantissa precision preserves attention quality
• Native Tensor Core support for FP8 GEMM

2.4 Accuracy Trade-offs: FP16 → FP8 → INT8 → INT4

Precision	Bytes/Elem	Compression vs FP16	Quality Impact (Perplexity)	Hardware Support
FP16 / BF16	2	1× (baseline)	Reference (0%)	Universal
FP8 (E4M3)	1	2×	< 0.05% (virtually lossless)	Hopper, Ada, Blackwell
INT8	1	2×	~0.1–0.3% (near-lossless)	Universal (all GPUs)
INT4	0.5	4×	1–5% (measurable; needs outlier protection)	Requires INT4 Tensor Cores (Blackwell, newer)

INT4 caveat: Naive INT4 quantization degrades math/code reasoning noticeably. Production systems use outlier-aware quantization (separate high-precision storage for outlier channels) or mixed precision (FP8 for Keys, INT4 for Values).

---

3. The Three-Tier Memory Hierarchy

When quantization alone is not enough, you move data across the hardware memory hierarchy:

Tier	Memory Type	Typical Capacity	Sustained Bandwidth	Latency (vs HBM)	Cost per GB
1	GPU HBM (HBM2e / HBM3)	80–96 GiB (H100: 80; H200: 141; MI300X: 192)	1,500–3,300 GB/s	1× (baseline)	~$30–50
2	CPU DDR5 RAM (System Memory)	256–2,048 GiB	200–400 GB/s	~60× slower transfer (PCIe Gen4 x16 ~25 GB/s)	~$2–4
3	Local NVMe SSD	4–16 TiB	3–7 GB/s (read)	~300× slower transfer	~$0.05–0.10

3.1 Bandwidth and Latency Reality

GPU HBM → GPU Compute: Direct, no transfer needed. Bandwidth = 1,500–3,300 GB/s.

GPU HBM ↔ CPU DDR5 (PCIe Gen4 x16): Must cross PCIe bus. Sustained throughput ≈ 25 GB/s (theoretical 31.5 GB/s). Latency overhead per transfer ≈ 5–10 µs + data time.

GPU HBM ↔ NVMe SSD (PCIe Gen4 x4): Must cross PCIe to CPU, then CPU to NVMe. Sustained read ≈ 5 GB/s. Latency overhead ≈ 20–50 µs + data time.

Key ratio: HBM bandwidth is 60× PCIe and 300× NVMe. A 4 GiB cache loads in 2.9 ms from HBM, 172 ms from CPU RAM, 859 ms from NVMe.

---

4. Pipelining and Latency Overlap

Offloading cannot be synchronous. Waiting 170 ms for a PCIe transfer at every decode step would make the system 10–100× slower.

Pipelined offloading pattern:

Time →
Decode Step t:     [GPU Compute] ──────────────────────→ logits_t
                        │
                        │  Async DMA: Prefetch blocks for step t+1
                        ▼
Prefetch for t+1:  [PCIe Transfer: CPU RAM → GPU HBM] ──→ blocks ready

• While GPU computes attention for token t, asynchronous DMA transfers the KV blocks needed for token t+1 from CPU RAM → GPU HBM.
• Requires double-buffering (two sets of GPU buffers) so compute and transfer never contend for the same memory.
• Engines like FlexGen (Sheng et al., 2023) and vLLM’s upcoming offload support implement this pattern.

Rule of thumb: If PCIe transfer time for the next step’s blocks < GPU compute time for current step, offloading is hidden (zero added latency). If transfer > compute, decode becomes transfer-bound.

---

5. Three Hands-On Exercises with Checked Solutions

Notation used below:

• 1 KiB = 1,024 bytes
• 1 MiB = 1,024 KiB
• 1 GiB = 1,024 MiB
• FLOPs = floating-point operations (total count)
• FLOPS = floating-point operations per second (rate)

---

Exercise 1: KV Cache INT8 Quantization Numerical Worked Example

Given: A 4-element Key vector k = [1.20, -0.45, 3.80, -2.10] in FP16. Quantize to signed INT8 (q_max = 127) using symmetric per-token scaling.

Step 1: Find absolute maximum

x_max = max(|1.20|, |-0.45|, |3.80|, |-2.10|) = 3.80

Step 2: Compute scale factor

s = x_max / q_max = 3.80 / 127 ≈ 0.0299212598

(Store s as FP16 or FP32 alongside the quantized values.)

Step 3: Quantize each element

Formula: q_i = round(x_i / s), clipped to [-127, 127].

q_0 = round(1.20 / 0.029921) = round(40.104) = 40
q_1 = round(-0.45 / 0.029921) = round(-15.039) = -15
q_2 = round(3.80 / 0.029921) = round(127.000) = 127   (hits q_max exactly)
q_3 = round(-2.10 / 0.029921) = round(-70.185) = -70

Quantized vector (stored in cache): k_q = [40, -15, 127, -70]
Storage: 4 bytes (INT8) vs original 8 bytes (FP16) → 2× compression.

Step 4: Dequantize and measure error

Formula: x̂_i = q_i * s

x̂_0 = 40 * 0.029921 = 1.19684
x̂_1 = -15 * 0.029921 = -0.44882
x̂_2 = 127 * 0.029921 = 3.80000
x̂_3 = -70 * 0.029921 = -2.09447

Reconstruction errors:

Index	Original	Reconstructed	Absolute Error	Relative Error
0	1.20	1.19684	0.00316	0.26%
1	-0.45	-0.44882	0.00118	0.26%
2	3.80	3.80000	0.00000	0.00%
3	-2.10	-2.09447	0.00553	0.26%

Max absolute error = 0.00553 (0.15% of dynamic range 3.80).
Max relative error = 0.26%.

Conclusion: INT8 with per-token scaling keeps error below 0.3% for typical KV values.

---

Exercise 2: Bandwidth and Latency Across the 3 Storage Tiers

Task: Load a 4 GiB KV cache buffer (4,294,967,296 bytes ≈ 4.295 GB) from each tier.

Tier	Sustained Bandwidth	Calculation	Transfer Time
GPU HBM	1,500 GB/s	4.295 / 1,500	2.86 ms
CPU DDR5 via PCIe Gen4 x16	25 GB/s	4.295 / 25	171.8 ms
NVMe SSD via PCIe Gen4 x4	5 GB/s	4.295 / 5	859.0 ms

Ratios vs HBM:

• PCIe transfer is 60× slower than HBM (171.8 / 2.86 ≈ 60)
• NVMe transfer is 300× slower than HBM (859.0 / 2.86 ≈ 300)

Practical implication: At 25 tokens/s decode throughput, each step has 40 ms budget. PCIe transfer of 4 GiB exceeds the budget by 4×. You must either:

• Transfer smaller blocks (pipelined, overlapping with compute)
• Keep more cache on GPU (quantize to fit)
• Accept lower throughput

---

Exercise 3: Combined Serving Strategy Under Strict VRAM Budget

Scenario: Serve 16 concurrent requests of Llama 3 8B at S = 16,384 (16K context).

Model dimensions:

• Layers L = 32
• KV heads H_kv = 8 (GQA: 32 query heads, 8 KV heads)
• Head dimension d_head = 128

Per-token KV cache size (from Lesson 16 formula):

bytes_per_token = 2 * L * H_kv * d_head * b
                = 2 * 32 * 8 * 128 * b
                = 65,536 * b bytes

At FP16 (b = 2): 128 KiB per token (131,072 bytes). At FP8/INT8 (b = 1): 64 KiB per token. At INT4 (b = 0.5): 32 KiB per token.

Part A: Total KV Cache Footprint

Precision	Bytes/Token	Per Request (16K tokens)	16 Requests Total
FP16	2	2.0 GiB	32.0 GiB
FP8 / INT8	1	1.0 GiB	16.0 GiB
INT4	0.5	0.5 GiB	8.0 GiB

Part B: Hardware Constraint

GPU has only 4.0 GiB VRAM available for KV cache (rest used by model weights, activations, workspace).

Question: Design a hybrid strategy using quantization + CPU offloading to fit 16 requests.

Solution Strategy

1. Apply FP8 quantization (reduces total footprint from 32 GiB → 16 GiB, 2× savings).
2. Keep 4.0 GiB of active working context on GPU HBM (fills available VRAM).
3. Offload remaining 12.0 GiB to CPU host RAM (DDR5, 25 GB/s PCIe).

Active window on GPU: 4.0 GiB / 64 KiB = 65,536 tokens total across all requests. With 16 requests: 4,096 tokens per request on GPU (25% of 16K context).

Offloaded to CPU: 12,288 tokens per request (75% of context).

Part C: Bandwidth Check for Sustained Throughput

Target: 25 tokens/s aggregate generation (all 16 requests combined).

Each new token requires:

• Reading the new token’s K/V from GPU HBM (already there for active window)
• For offloaded tokens: When attention reaches into the offloaded region, those blocks must be prefetched.

Assume sliding attention window of 4,096 tokens (matches GPU-resident portion). For tokens beyond 4K, the engine must stream blocks from CPU RAM.

Worst-case PCIe bandwidth demand:

• Each decode step reads 1 token’s KV from each of 32 layers.
• Offloaded portion: 12,288 tokens * 64 KiB = 768 MiB per request.
• But with pipelining, only the next needed blocks transfer.

Required streaming bandwidth:

Active working set on GPU = 4,096 tokens per request
At 25 tokens/s aggregate → each request generates ~1.56 tokens/s
Time per token per request = 0.64 s
Bytes per token per layer = 2 × 8 heads × 128 elements × 1 byte = 2 KiB
Total across all 32 layers = 32 × 2 KiB = 64 KiB per token

PCIe bandwidth needed = 64 KiB/token × 25 tokens/s
                     = 1,600 KiB/s = 1.5625 MiB/s ≈ 0.0016 GB/s

Result: ~0.0016 GB/s required << 25 GB/s available PCIe bandwidth. Even with practical bus overhead and chunked prefetching, this consumes less than 0.1% of available PCIe bandwidth. Comfortable margin.

Conclusion: FP8 quantization + CPU offloading fits 16× 16K requests in 4 GiB VRAM with ample PCIe headroom for pipelined prefetch.

---

6. Summary Reference Table

Dimension	FP16 / BF16	FP8 (E4M3)	INT8	INT4	CPU Offload (DDR5)	NVMe Offload
Bytes per element	2	1	1	0.5	1–2 (same format)	1–2
Compression vs FP16	1×	2×	2×	4×	N/A (capacity)	N/A (capacity)
Quality impact	Reference	< 0.05%	~0.1–0.3%	1–5%*	None	None
Compute for dequant	None	FP8→FP16 (Tensor Core)	INT8→FP16 (Tensor Core)	INT4→FP16 (needs INT4 TC)	PCIe DMA + dequant	PCIe + NVMe + dequant
GPU VRAM needed	Baseline	50%	50%	25%	Near-zero (active only)	Near-zero
Transfer latency (4 GiB)	N/A	N/A	N/A	N/A	172 ms (PCIe Gen4)	859 ms (NVMe)
Hardware requirement	All GPUs	Hopper/Ada/Blackwell	All GPUs	Blackwell / INT4 TC	Any PCIe Gen4+	Any NVMe
Best use case	Max quality, VRAM fits	Default for H100+/RTX 40	Default for older GPUs	Extreme compression	VRAM exhausted, host RAM available	Massive context, cost-sensitive

*INT4 requires outlier protection (separate FP16 storage for outlier channels) to stay near 1% quality loss.

---

7. Primary Source Citations

#	Citation	Link
1	Dettmers et al. (2022), LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale	arXiv:2208.07339
2	Sheng et al. (2023), FlexGen: High-throughput Generative Inference of Large Language Models with a Single GPU	arXiv:2303.04665
3	Frantar et al. (2022), GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers	arXiv:2210.17323
4	Hooper et al. (2024), KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization	arXiv:2401.18079
5	Xiao et al. (2023), StreamingLLM: Efficient Streaming Language Models with Attention Sinks	arXiv:2309.17453

---

8. What Comes Next

• Next lesson: Chunked Prefill and Long-Context Limits — Now that we can compress and offload the cache, learn how to split long prompt processing into scheduled pieces and examine why memory, computation, and useful recall become harder as context grows.
• Previous lesson: Reuse, Evict, or Limit Earlier Context
• Track overview: Start Here: LLM Foundations