Module 04 · Chapter 14

Prefill vs. Decode: A Worked Latency Example

Compare phase costs and tensor shapes, then calculate how prompt and output lengths change total request latency.

Intermediate14 min readUpdated Nov 8, 2026

Learning objectives

Compare the mathematical equations, tensor shapes, and operational characteristics of prefill and decode side by side
Formulate the end-to-end latency equation combining TTFT and generation time
Calculate prefill FLOPs, decode memory reads, TTFT, and ITL for a 7B parameter model
Identify the crossover point where decode latency overtakes prefill latency
Analyze how different workloads (summarization, code generation, reasoning) shift the serving bottleneck
Evaluate latency optimization strategies based on prefill versus decode dominance

Before you start

Decode: Generate One Position at a Time (/llm-foundations/decode-reuse-the-cache/)

In Prefill: Process the Prompt and Build the Cache and Decode: Generate One Position at a Time, you traced each phase independently. Now we bring them together — side by side — to see how their costs combine into the total latency a user experiences. This lesson gives you the mental model and the calculator you need to answer: “Why does my chat request take 2 seconds but my summarization request takes 1.5 seconds even with a 4x longer prompt?”

Side-by-Side Comparison of the Two Phases

Before we build the latency equation, let’s contrast prefill and decode across every dimension that matters for serving.

Tensor Shapes and Operations

Aspect	Prefill (Phase 1)	Decode (Phase 2)
Input tensor	`[B, P, d_model]`	`[B, 1, d_model]`
Query shape	`[P, d_k]`	`[1, d_k]`
Key shape	`[P, d_k]`	`[1, d_k]` (new) + cache `[T_past, d_k]`
Value shape	`[P, d_v]`	`[1, d_v]` (new) + cache `[T_past, d_v]`
Attention matrix	`[P, P]` square causal	`[1, T_past+1]` row vector
Core operation	Matrix-Matrix GEMM	Matrix-Vector GEMV
Primary bottleneck	Compute (FLOPS-bound)	Memory bandwidth (HBM-bound)
Arithmetic intensity	~1000 FLOPs/byte	~1 FLOP/byte (B=1)
KV cache action	Create + write initial	Read existing + append 1 entry
Causal mask	Required (lower-triangular)	Not needed (all past is valid)
Latency metric	Time to First Token (TTFT)	Inter-Token Latency (ITL)

Why the shapes matter: Prefill processes P tokens in one giant matrix multiply. Decode processes exactly one token per step. The attention matrix collapses from P×P to 1×(T_past+1) — a row vector. This shape change is the root cause of the arithmetic intensity collapse.

Hardware Bottleneck: The Root Cause

Prefill (Compute-Bound):

Weights read once, reused across P tokens
Large GEMMs saturate Tensor Cores (e.g., 150 TFLOPS achieved on H100)
Time ∝ P × N / FLOPS

Decode (Memory-Bound):

Weights read every token (14 GB for 7B FP16)
Single-token GEMV cannot reuse weights across sequence
Time ∝ Weight_Bytes / Bandwidth

The Serving Paradox

This is the most counterintuitive fact in LLM serving, and it determines every optimization decision you will make:

Prefill performs over 90% of the total mathematical operations (FLOPs) of a request, but typically finishes in just a few percent of the total time.

Decode performs less than 10% of the FLOPs, but consumes 90% or more of the user-experienced wall-clock latency!

Why?

Because prefill runs at high compute utilization (e.g., 150 TFLOPS on H100), whereas single-token decode is starved for memory bandwidth (e.g., 800 GB/s loading 14 GB of weights for every 14 GFLOPs of math).

Let’s verify with numbers for a 7B model, P=1024, G=128:

Prefill FLOPs = 2 × P × N = 2 × 1024 × 7×10⁹ = 14.336 TFLOPs
Prefill time = 14.336 TFLOPs / 150 TFLOPS = 95.6 ms

Decode per token: load 14 GB weights at 800 GB/s = 17.5 ms
Total decode time = 128 × 17.5 ms = 2240 ms = 2.24 s

Result: Prefill = 95.6 ms (4.1%), Decode = 2240 ms (95.9%).

The same model weights that took 95 ms to process 1024 tokens in parallel take 2.24 s to process 128 tokens one by one.

The Unified End-to-End Latency Equation

The total request time is the sum of prefill time (Time to First Token) and decode time (generation time).

T_total = TTFT + (G - 1) × ITL

Where:

TTFT = Time to First Token (includes network overhead, tokenization, and prefill compute)
G = total number of generated tokens
ITL = Inter-Token Latency (time per decode step)

Important distinction: The first token is emitted at TTFT. The remaining G-1 tokens each take one ITL. Some presentations simplify to T_total = TTFT + G × ITL by absorbing the first token’s decode step into TTFT. Both are correct if you define the terms consistently. We use the explicit (G - 1) form to keep TTFT and ITL as independent measurements.

Prefill Time (TTFT)

TTFT = T_overhead + Prefill_Compute_Time
     = T_overhead + (2 × P × N) / Achieved_FLOPS

T_overhead = network + tokenization + kernel launch (typically 5–20 ms)
P = prompt length in tokens
N = model parameters
Achieved_FLOPS = GPU peak FLOPS × MFU (Model Flops Utilization)

Decode Time (ITL)

ITL = (Model_Weight_Bytes + KV_Cache_Bytes_Per_Token) / Achieved_Memory_Bandwidth

Model_Weight_Bytes = total model size (14 GB for 7B FP16)
KV_Cache_Bytes_Per_Token = 2 × L × H_kv × d_head × bytes_per_element
Achieved_Memory_Bandwidth = realistic HBM throughput (e.g., 800 GB/s on H100)

For short-to-medium contexts where the KV cache fits in L2, the cache term is negligible and:

ITL ≈ Model_Weight_Bytes / Achieved_Memory_Bandwidth

The Crossover Point

When does prefill compute time equal decode time? Set the two dominant terms equal:

(2 × P × N) / Achieved_FLOPS = G × (Model_Weight_Bytes / Achieved_Bandwidth)

Solve for the ratio P / G:

P / G = (Achieved_FLOPS / Achieved_Bandwidth) × (Model_Weight_Bytes / (2 × N))

For a 7B model (FP16 = 14 GB weights) on hardware achieving 150 TFLOPS compute and 800 GB/s bandwidth:

P / G = (150×10¹² / 800×10⁹) × (14×10⁹ / (2 × 7×10⁹))
      = (187.5) × (14 / 14)
      = 187.5

What This Means

Workload Type	Typical `P / G`	Dominant Phase	Example
Summarization	100–200	~Balanced or prefill-heavy	`P=4000, G=50` → ratio 80
Chat / Q&A	5–20	Decode-heavy	`P=1024, G=128` → ratio 8
Code Generation	1–5	Strongly decode-heavy	`P=512, G=512` → ratio 1
Reasoning / Chain-of-Thought	1–3	Extremely decode-heavy	`P=400, G=400` → ratio 1
Massive Doc Q&A	200–500+	Prefill-heavy	`P=8000, G=40` → ratio 200

Rule of thumb: If P / G < 187.5 (the vast majority of chat, coding, and reasoning tasks), decode dominates total latency. If P / G > 187.5 (massive document summarization, long-context RAG), prefill dominates.

Hands-On Exercise 1: End-to-End Latency and Breakdown for a Chat Request

Problem: Calculate the complete latency breakdown for a typical chat request.

Given:

Model: 7B parameters (N = 7 × 10⁹), FP16 = 14 GB weights
Hardware: 150 TFLOPS achieved compute, 800 GB/s achieved bandwidth
Fixed overhead: T_overhead = 15 ms
Workload: P = 1024 prompt tokens, G = 128 generated tokens

Step-by-step Solution:

Step 1: Prefill FLOPs

Prefill FLOPs = 2 × P × N
              = 2 × 1024 × 7 × 10⁹
              = 14.336 × 10¹²
              = 14.336 TFLOPs

Step 2: Prefill compute time

Prefill_Time = Prefill_FLOPs / Achieved_FLOPS
             = 14.336 TFLOPs / 150 TFLOPS
             = 0.09557 s
             = 95.6 ms

Step 3: TTFT

TTFT = T_overhead + Prefill_Time
     = 15 ms + 95.6 ms
     = 110.6 ms

Step 4: Decode ITL (memory-limited)

ITL = Model_Weight_Bytes / Achieved_Bandwidth
    = 14 GB / 800 GB/s
    = 0.0175 s
    = 17.5 ms/token

Step 5: Total decode time

Decode_Time = G × ITL
            = 128 × 17.5 ms
            = 2240 ms
            = 2.24 s

Step 6: Total request time

T_total = TTFT + Decode_Time
        = 110.6 ms + 2240 ms
        = 2350.6 ms
        ≈ 2.35 s

Step 7: Percentage breakdown

Prefill share = TTFT / T_total = 110.6 / 2350.6 ≈ 4.7%
Decode share  = Decode_Time / T_total = 2240 / 2350.6 ≈ 95.3%

Step 8: Effective generation throughput

Effective_TPS = G / T_total
              = 128 / 2.3506
              ≈ 54.5 tokens/second

Final Answer: Total latency ≈ 2.35 s. Prefill contributes 4.7%, decode contributes 95.3%. Effective throughput = 54.5 tok/s (vs. ideal 57.1 tok/s if prefill were free).

Hands-On Exercise 2: Workload Contrast — Summarization vs. Code Generation

Problem: Compare two workloads with the same model and hardware to see how the bottleneck shifts.

Shared Given:

Model: 7B, FP16 = 14 GB weights
Hardware: 150 TFLOPS compute, 800 GB/s bandwidth, 15 ms overhead
ITL = 17.5 ms/token (from Exercise 1)

Workload A: Summarization (`P = 4096, G = 64`)

Prefill compute time:

Prefill_FLOPs = 2 × 4096 × 7 × 10⁹ = 57.344 TFLOPs
Prefill_Time  = 57.344 / 150 = 0.3823 s = 382.3 ms

TTFT:

TTFT = 15 ms + 382.3 ms = 397.3 ms

Decode time:

Decode_Time = 64 × 17.5 ms = 1120.0 ms = 1.12 s

Total time:

T_total = 397.3 ms + 1120.0 ms = 1517.3 ms ≈ 1.52 s

Breakdown:

Prefill share = 397.3 / 1517.3 = 26.2%
Decode share  = 1120.0 / 1517.3 = 73.8%
Effective TPS = 64 / 1.5173 ≈ 42.2 tok/s

Workload B: Code Generation (`P = 512, G = 512`)

Prefill compute time:

Prefill_FLOPs = 2 × 512 × 7 × 10⁹ = 7.168 TFLOPs
Prefill_Time  = 7.168 / 150 = 0.0478 s = 47.8 ms

TTFT:

TTFT = 15 ms + 47.8 ms = 62.8 ms

Decode time:

Decode_Time = 512 × 17.5 ms = 8960.0 ms = 8.96 s

Total time:

T_total = 62.8 ms + 8960.0 ms = 9022.8 ms ≈ 9.02 s

Breakdown:

Prefill share = 62.8 / 9022.8 = 0.7%
Decode share  = 8960.0 / 9022.8 = 99.3%
Effective TPS = 512 / 9.0228 ≈ 56.7 tok/s

Comparison Table

Metric	Summarization (A)	Code Generation (B)
Prompt tokens (P)	4096	512
Generated tokens (G)	64	512
P / G ratio	64	1
TTFT	397 ms	63 ms
Decode time	1.12 s	8.96 s
Total latency	1.52 s	9.02 s
Prefill % of total	26.2%	0.7%
Decode % of total	73.8%	99.3%
Effective TPS	42.2 tok/s	56.7 tok/s
Dominant bottleneck	Mixed (decode-leaning)	Decode (memory bandwidth)

Key insight: Even though Summarization (A) has 8× more prompt tokens, it finishes 6× faster because it generates far fewer tokens. Code Generation (B) spends 99% of its time waiting for memory bandwidth.

Hands-On Exercise 3: Module Exit Task — Part IV Capstone Serving Decision

Problem: You are the serving engineer for two production applications. For each, calculate the latency breakdown and recommend the top three optimization priorities with justification.

Shared Given:

Model: 7B, FP16 = 14 GB weights
Hardware: 150 TFLOPS compute, 800 GB/s bandwidth, 15 ms overhead
ITL = 17.5 ms/token

Application 1: Long Document Q&A (`P = 8000, G = 100`)

Step 1: Prefill compute time

Prefill_FLOPs = 2 × 8000 × 7 × 10⁹ = 112 TFLOPs
Prefill_Time  = 112 / 150 = 0.7467 s = 746.7 ms

Step 2: TTFT

TTFT = 15 ms + 746.7 ms = 761.7 ms

Step 3: Decode time

Decode_Time = 100 × 17.5 ms = 1750.0 ms = 1.75 s

Step 4: Total

T_total = 761.7 ms + 1750.0 ms = 2511.7 ms ≈ 2.51 s

Step 5: Breakdown

Prefill share = 761.7 / 2511.7 = 30.3%
Decode share  = 1750.0 / 2511.7 = 69.7%

Engineering Recommendation — Priority Order:

Chunked Prefill (Sarathi / vLLM chunked prefill): Prefill takes 747 ms and blocks decode for other requests. Split the 8000-token prefill into chunks (e.g., 512 tokens) so decode steps from other requests can interleave. Prevents head-of-line blocking.
Prefix Caching: If the same long document is queried repeatedly (common in RAG), cache the prompt’s KV cache across requests. Avoids recomputing the 747 ms prefill entirely for repeat queries.
FlashAttention-2 / FlashAttention-3: The prefill attention is O(P²) = 64M elements. FlashAttention reduces HBM reads/writes during the Q@K^T and softmax@V steps, directly cutting the 747 ms prefill time. Also reduces peak memory pressure.

Why not speculative decoding? Decode is only 70% of latency here. Speculative decoding helps decode, but prefill is the larger optimizable component. Fix prefill first.

Application 2: Autonomous Agent Loop (`P = 400, G = 400`)