Learn · LLM inference
Inside a model, level by level
What the model computes, from tokens down to the loops a GPU runs. Every level is a diagram from another page, so this page and those pages always show the same picture.
Map · 29 levels
The 36 layers have the same shape and run one after another. Prefill runs the stack once over the whole prompt; decode runs it once per new token and reuses the KV cache each layer wrote.
Most of the time goes to the GEMMs and attention; norms and RoPE are small kernels that mostly move memory. In decode every weight is read from HBM to make one token per sequence, so the GEMMs wait on memory, and batching more sequences is almost free.
Every level of tiling exists for reuse: a tile read once from HBM feeds many multiply-adds from LDS and registers. Bigger tiles mean more reuse but fewer workgroups to keep all CUs busy.
Two LDS buffers let the next tile load while the matrix cores work on the current one, so neither side waits.
Only the new token is projected; the keys and values of all earlier tokens are read from the cache. That read grows with the context, so long contexts make decode attention wait on HBM.
Blocks let a sequence grow without reserving memory for the longest possible reply, and let requests with the same prefix share the same blocks.
The score matrix never goes to HBM: it is made one block at a time in registers, with a running max and sum that keep the softmax exact. Scores, softmax and the weighted sum are one kernel.
One token reads only 4 experts' weights. A batch is different: its tokens pick different experts, so most experts are read anyway.
How to read it
- Click a box with the badge to open it below; click it again, or press Esc, to close it. The breadcrumb goes back up; Map lists every level.
- Boxes that lead to the same thing open the same panel: attention and MoE both open the layer’s kernels; every GEMM opens one GEMM view.
Where each level comes from
| Level | Page |
|---|---|
| The forward pass | Tokens and transformers |
| One layer as kernels | Prefill and decode kernels |
| Attention and the KV cache | Attention and the KV cache, KV cache management |
| FlashAttention | FlashAttention |
| The router | Mixture of experts |
| A GEMM | GEMM tiling |