How LLM inference works
An animated explainer of what happens when you send a prompt to a large language model: tokens, routing, the forward pass and streaming the answer back.
Chapters
- Overview: A prompt becomes tokens, gets routed to a GPU replica, scheduled into a batch, and generated one token per forward pass, streamed back as it goes.
- Routing: The router hashes the system prompt so requests land where their prefix is already cached, spilling to the least-loaded replica when queues grow.
- Continuous Batching: The scheduler works per iteration. When a sequence hits end-of-sequence, its slot is refilled on the very next step. The GPU never waits.
- Paged KV Cache: Keys and values live in fixed 16-token pages. A block table maps logical blocks to physical pages, so memory grows on demand and prefixes are shared.
- GPU Kernels: Each step runs 32 layers of GEMMs and paged attention. Decode is memory-bound, so every step streams all the weights. Batching amortizes that read.
- Sampling & Streaming: Logits are sampled into a token, detokenized incrementally, and pushed over server-sent events. Users feel time to first token and inter-token latency.
Make an animated explainer of your own topic →