How LLM inference works

An animated explainer of what happens when you send a prompt to a large language model: tokens, routing, the forward pass and streaming the answer back.

Chapters

  1. Overview: A prompt becomes tokens, gets routed to a GPU replica, scheduled into a batch, and generated one token per forward pass, streamed back as it goes.
  2. Routing: The router hashes the system prompt so requests land where their prefix is already cached, spilling to the least-loaded replica when queues grow.
  3. Continuous Batching: The scheduler works per iteration. When a sequence hits end-of-sequence, its slot is refilled on the very next step. The GPU never waits.
  4. Paged KV Cache: Keys and values live in fixed 16-token pages. A block table maps logical blocks to physical pages, so memory grows on demand and prefixes are shared.
  5. GPU Kernels: Each step runs 32 layers of GEMMs and paged attention. Decode is memory-bound, so every step streams all the weights. Batching amortizes that read.
  6. Sampling & Streaming: Logits are sampled into a token, detokenized incrementally, and pushed over server-sent events. Users feel time to first token and inter-token latency.

Make an animated explainer of your own topic →

More explainers

← All explainers

Try AI Flowchart free →