Inference Optimisation & AI Infrastructure

What you'll be able to do

By the end of the core course.

1. Separate prefill and decode costs when diagnosing latency
2. Estimate KV cache memory for a model, context, and batch size
3. Choose batching, paging, and quantisation strategies against a target
4. Plan GPU capacity from a traffic profile and a latency budget
5. Build and benchmark a small serving system, then defend its architecture under load and failure

Inference Optimisation & AI Infrastructure

In progress · 4 modules live

Read the modules in order. Each card links to the module's first lesson and states the reader question it answers and the work product you produce.

1. Module 013 lessons live
   Measure and Explain Inference Performance

   Reader question: "Why is my first token or stream slow?"

   Work product: Latency and Memory Attribution Report.

   Start with: Serving Workloads and Performance Metrics

2. Module 024 lessons live
   Schedule Requests and Manage KV Memory

   Reader question: "How do we keep the GPU saturated without running out of memory?"

   Work product: Continuous Batching and Chunk Budget Policy.

   Start with: Batching and throughput

3. Module 034 lessons live
   Quantize Without Losing the Result

   Reader question: "Can we cut weight and KV memory without degrading accuracy?"

   Work product: Hardware-Calibrated Precision Gate.

   Start with: Quantization: Scale, Zero Point, and Granularity

4. Module 045 lessons live
   Build a Serving System

   Reader question: "How many GPUs do we need and what happens when traffic spikes?"

   Work product: Production Capacity and Cost Decision Record.

   Start with: Speculative Decoding

Main subjects

The territory this track covers.

1. 01Part VII — Attention optimization and efficient kernels
2. 02Part VIII — Batching and inference scheduling
3. 03Part IX — Model compression and faster decoding
4. 04Part X — Multi-GPU and distributed inference
5. 05Part XI — Production inference engines
6. 06Part XII — Performance engineering and capacity planning
7. 07Part XIII — Hands-on implementation
8. 08Part XIV — Advanced systems design and engineering trade-offs

Who this is for

Engineers who run models in production or on their own hardware and need to defend the cost and latency numbers.

Before you start

• New to GPU programming: start with Track 00 Part VI, GPU Fundamentals for Inference
• Read How LLM inference works, or know the difference between prefill and decode
• Comfort with multiplication and powers of two
• Some experience running a model or using a hosted inference API

Curriculum

16 of 35 lessons published

Core course

16 of 16 live

The main sequence, read in order.

Module 01

Measure and Explain Inference Performance

How to instrument and explain inference latency and memory before changing any system.

3 of 3 lessons live

1. 01Serving Workloads and Performance MetricsDefine TTFT, inter-token latency, throughput, and tail latency, then map each to a reproducible measurement and the instrument that actually answers it.Intermediate12 min readRead
2. 02Where Inference Time and Memory GoBreak down inference cost into weights, KV cache, compute, and memory bandwidth, and identify whether prefill or decode is the bottleneck for a stated workload.Intermediate13 min readRead
3. 03FlashAttention and GPU Data MovementExplain why reducing HBM reads can matter more than reducing FLOPs, and trace where FlashAttention avoids materialising the full attention matrix.Advanced16 min readRead

Module 02

Schedule Requests and Manage KV Memory

How schedulers and cache managers keep the GPU busy without running out of memory.

4 of 4 lessons live

1. 04Batching and ThroughputAnalyze how batching amortizes weight transfers in LLM decode, derive memory-versus-compute bounds with KV traffic, and evaluate queue and capacity tradeoffs.Intermediate26 min readRead
2. 05PagedAttention: Virtual Memory for the KV CacheMap logical KV positions to physical blocks, trace copy-on-write sharing, and calculate capacity and tail waste without assuming memory savings guarantee throughput.Advanced35 min readRead
3. 06Continuous Batching: Iteration-Level SchedulingTrace iteration-level scheduling, request state machines, packed tensor representations, roofline compute-memory dynamics, and KV memory budget limits in LLM serving engines.Advanced30 min readRead
4. 07Prefill Optimisation: Prefix Caching, Chunked Prefill, and DisaggregationPrefix caching reuses KV blocks across queries to skip redundant prefill compute, while chunked prefill bounds per-iteration tokens to protect decode latency. We trace both mechanisms and tradeoffs.Advanced35 min readRead

Module 03

Quantize Without Losing the Result

How precision choices change memory, speed, and quality, and how to verify them.

4 of 4 lessons live

1. 08Quantization: Scale, Zero Point, and GranularityChoose the quantization granularity (per-tensor, per-channel, per-group) and measure the rounding error it introduces on a weight matrix.Advanced15 min readRead
2. 09Calibration Data and Quality RegressionPick representative calibration data, run post-training quantization, and decide whether the quality drop on a held-out set is acceptable before deploying.Advanced18 min readRead
3. 10Low-bit LLM Weights and ActivationsSeparate what is quantized (weights, activations, KV cache) from the numeric format (INT8, FP8, 4-bit), then compare each combination against hardware support and quality tradeoffs.Advanced15 min readRead
4. 11KV-Cache QuantizationMeasure memory saved by quantizing the KV cache, the quality impact on a target task, and the runtime cost of dequantisation.Advanced14 min readRead

Module 04

Build a Serving System

How to assemble precision, scheduling, and capacity into a defensible serving design.

5 of 5 lessons live

1. 12Speculative DecodingExplain draft-and-verify, estimate when it reduces generation latency, and identify the conditions where it does not help.Advanced15 min readRead
2. 13Choose a Serving Runtime and HardwareCompare CPU, single-GPU, and production serving paths using the same workload, and state the constraints that decide between them.Advanced15 min readRead
3. 14Capacity Planning: From Traffic Profile to GPU CountTurn a latency target and a traffic profile into a GPU count you can defend, using memory bounds, queueing theory, and a load-sweep protocol.Advanced18 min readRead
4. 15Scale Beyond One GPU: Replication, Tensor Parallelism, and Prefill/Decode DisaggregationChoose a multi-GPU serving topology—replicas, tensor parallelism, or prefill/decode disaggregation—given a model footprint, traffic target, latency SLOs, and GPU interconnect.Advanced16 min readRead
5. 16Overload and Failure HandlingWhen a burst exceeds serving capacity, decide which requests to admit, reject, cancel, or route to a fallback while protecting latency, useful throughput, quality, and cost.Advanced16 min readRead

Advanced modules

0 of 19 live

Deep dives for readers who have finished the core course.

Module 05

Part VII — Attention Optimization and Efficient Kernels

Build on the existing FlashAttention lesson. Compare ordinary attention with tiled, IO-aware attention, online softmax, fusion, and decode-specific kernels. Exit task: implement a tiny tiled attention calculation and explain when it changes memory traffic without claiming it always changes arithmetic complexity.

0 of 1 lessons live

1. 17Implement Tiled Attention and Online SoftmaxWrite a small tile-by-tile attention calculation, compare it with ordinary attention, and explain FlashDecoding and prefill/decode differences.AdvancedPlannedComing soon

Module 06

Part VIII — Batching and Inference Scheduling

Extend the existing batching and cache lessons to queues, sequence-level decisions, prefill/decode interference, admission, priorities, cancellation, fairness, tail latency, and separated prefill/decode workers. Exit task: walk a multi-request timeline and defend every scheduling choice.

0 of 2 lessons live

1. 18Queues, Priorities, Fairness, and CancellationTrace requests arriving at different times through static, dynamic, and continuous batching, including head-of-line blocking and cancellation.AdvancedPlannedComing soon
2. 19A Multi-Request Scheduler TimelineChoose work under token and KV-cache budgets; compare chunked prefill, priority rules, and separate prefill/decode workers.AdvancedPlannedComing soon

Module 07

Part IX — Model Compression and Faster Decoding

Build on quantization and speculative decoding to compare weight, activation, and cache precision; training-aware methods; draft/target acceptance; multiple-token proposals; early exits; expert routing; and small-model serving. Exit task: choose a technique only when its measured quality, memory, and latency meet a stated workload.

0 of 2 lessons live

1. 20GPTQ, AWQ, and Quantization-Aware TrainingCompare post-training and training-aware quantization of weights, activations, and cache; test accuracy, speed, and memory trade-offs.AdvancedPlannedComing soon
2. 21Beyond One-Token DecodingCompare speculative acceptance rules, Medusa-style token proposals, early exits, mixture-of-experts routing, and distilled small models.AdvancedPlannedComing soon

Module 08

Part X — Multi-GPU and Distributed Inference

Extend the existing scale-beyond-one-GPU lesson with model memory accounting, tensor/pipeline/data/expert parallelism, collectives, interconnects, placement, multi-node serving, disaggregation, and failure recovery. Exit task: compare GPU layouts for small and large models under communication and latency limits.

0 of 2 lessons live

1. 22Pipeline, Expert, and Replica ParallelismPlace model layers, experts, or full replicas across GPUs and compare memory, communication, and failure consequences.AdvancedPlannedComing soon
2. 23Collectives, Links, and Multi-Node PlacementCalculate all-reduce and all-gather traffic across NVLink, PCIe, and networks; compare tensor parallelism and prefill/decode separation.AdvancedPlannedComing soon

Module 09

Part XI — Production Inference Engines

Distinguish a serving engine from a model. Compare model loading, tokenization, request processing, scheduling, cache management, GPU execution, streaming, APIs, observability, deployment, hardware support, and limits using pinned official sources. Exit task: choose an engine for two stated workloads.

0 of 2 lessons live

1. 24Inside an Inference Engine RequestTrace a request through loading, tokenization, queueing, cache allocation, GPU execution, streaming, and monitoring.AdvancedPlannedComing soon
2. 25Choose an Inference Engine for a WorkloadCompare vLLM, SGLang, Hugging Face TGI, TensorRT-LLM, llama.cpp, Ollama, and significant newer engines using versioned evidence.AdvancedPlannedComing soon

Module 10

Part XII — Performance Engineering and Capacity Planning

Join the existing metrics, memory, and capacity lessons into one model with queueing, percentiles, GPU utilization, approximate compute/bandwidth bounds, and cost per request or input/output token. Exit task: estimate before deploying, then state what requires real benchmarking.

0 of 1 lessons live

1. 26A Serving Capacity and Cost WorkbookVary concurrency and prompt/output lengths; calculate memory, latency, throughput, queueing, and cost per input or output token.AdvancedPlannedComing soon

Module 11

Part XIII — Hands-On Implementation

Begin with a NumPy-only softmax, sampling, and attention project. Track 1 already has PyTorch decoder and cache labs for Projects 2–3. Continue with phase benchmarking, a scheduler simulator, a small-model server, a serving benchmark, and a production design. Each project needs prerequisites, runnable code, setup, expected observations, mistakes, and extensions; label GPU-only work.

0 of 6 lessons live

1. 27Project 1: Softmax, Sampling, and Attention in NumPyImplement the three small operations without PyTorch, compare numerical results with hand calculations, and inspect tensor shapes.BeginnerPlannedComing soon
2. 28Project 4: Benchmark Prefill and DecodeMeasure the two phases at different context lengths and compare observations with the simple latency model.IntermediatePlannedComing soon
3. 29Project 5: Simulate Multiple RequestsBuild a CPU-only request scheduler simulator and inspect queues, cache budgets, throughput, and tail latency.AdvancedPlannedComing soon
4. 30Project 6: Serve a Small Open-Weight ModelRun a small model in a suitable serving engine, issue requests, and inspect streaming and operational metrics.AdvancedPlannedComing soon
5. 31Project 7: Benchmark Serving ChoicesSweep batching, caching, precision, concurrency, and streaming with fixed workload and quality checks.AdvancedPlannedComing soon
6. 32Project 8: Design a Production Inference ServiceJoin latency targets, capacity estimates, failure policy, and observability in a reviewable service design.AdvancedPlannedComing soon

Module 12

Part XIV — Advanced Systems Design and Engineering Trade-offs

Use realistic scenarios with stated assumptions, compute/memory estimates, alternative architectures, scheduling and caching choices, cost/performance bounds, failure modes, and a justified recommendation. Never invent precise benchmark results. Exit task: defend a serving architecture and the measurements that could overturn it.

0 of 3 lessons live

1. 33Design for Model Size and Long PromptsWork scenarios for a 7B/8B model serving 1,000 users, a latency-constrained 70B model, and long prompts under limited GPU memory.AdvancedPlannedComing soon
2. 34Design for Cost, First-Token Speed, and Traffic SpikesCompare low-cost and low-first-token-latency designs; handle unpredictable traffic and isolation across tenants.AdvancedPlannedComing soon
3. 35Design for Separation, Deployment Choice, and OverloadCompare separate prefill/decode workers, self-hosting versus managed APIs, and a server that slows when concurrency rises.AdvancedPlannedComing soon

The core course is the shortest complete path. Advanced modules assume you have finished it.

Recommended start

Serving Workloads and Performance Metrics

Define TTFT, inter-token latency, throughput, and tail latency, then map each to a reproducible measurement and the instrument that actually answers it.

Start chapter 1

At a glance

Status

Live

Difficulty

Advanced

Level

Intermediate to advanced

Modules

12

Lessons

16 of 35 live

Reading time

309 min live