Learn · Kernels & silicon
Prefill and decode as GPU kernels
The pages before this one built up the pieces: how waves run, what instructions exist, why memory decides speed, GEMM tiling and FlashAttention. This page puts them together: what a GPU actually runs for one step of an LLM, and why prefill and decode behave so differently on the same hardware.
The idea first
What a “kernel” is here. The model is a recipe of about ten steps per layer: normalise, project, attend, project, route, run experts. The GPU runs each step as one or a few kernels (GPU functions launched over a grid of threads). The same kernels run in prefill and in decode. What changes is how many token rows go through each one at once.
The analogy: a heavy tool fetched from the warehouse. Each weight matrix is a heavy tool kept in HBM (the warehouse). Every step, the GPU carries each tool to the matrix cores (the workbench), because the full set (62.9 GB) is far bigger than any cache. Then it uses the tool on every row it has:
| Kernel kind | What it does to each token | The question it answers | Cost driver |
|---|---|---|---|
| GEMM (QKV, output, router, experts) | multiply the token’s vector by a weight matrix | “Transform this token” | the weights, read once per step for all rows |
| Attention | mix in earlier tokens’ keys and values | “What do earlier tokens tell me?” | the KV cache, per sequence |
| Element-wise (norm, RoPE, KV write) | tidy, rotate or copy each vector | “Prepare this token for the next step” | reading and writing the vectors |
- Prefill brings thousands of prompt tokens to the bench at once: one fetch of a tool, thousands of uses. The bench (math) is the bottleneck.
- Decode brings one new token per sequence: with a batch of 64, one fetch, 64 uses. The trips to the warehouse (HBM bandwidth) are the bottleneck.
- Fetch the toolEvery step reads each weight matrix from HBM once.
- Prefill: thousands of rows per fetchA 4,085-token prompt runs all its rows through each fetched matrix: about 4,000 uses per weight. The matrix cores are the limit.
- Decode: a few rows per fetch64 sequences give 64 uses per weight. The cores finish long before the next weights arrive; HBM bandwidth is the limit.
- Attention reads per-sequence memoryAttention reads each sequence’s own KV cache. Batching does not help here: one sequence’s cache is useless to another.
Where the analogy breaks. Caches do keep some data close (L2, Infinity Cache), and kernels overlap fetching with math, so real time is not a clean sum. And MoE splits the tools: 128 expert matrices, 4 used per token, so in decode most experts get one or two tokens each.
One layer, as kernels
The example is gpt-oss-120b, a public open-weight mixture-of-experts model. Shapes from its published config: hidden size 2,880; 36 layers (alternating sliding-window and full attention); 64 query heads and 8 KV heads of size 64; 128 experts, 4 per token, expert width 2,880. Pick a phase, then step through the layer: each box shows that kernel’s intensity, its roofline time bound on one MI355X, and what limits it (green: compute, amber: memory).
- RMSNormRMSNorm, 4,085 rows × 2,880: 47.1 MFLOP, 1.0 FLOP/B against a ridge of ~313, so memory-bound: reading 47.1 MB takes 5.9 µs, the math only 0.0 µs.
- QKV projectionQKV projection, [4,085 × 2,880] · [2,880 × 5,120]: 120 GFLOP, 1,270 FLOP/B against a ridge of ~313, so compute-bound: the matrix cores need 48.2 µs, more than the 11.9 µs to read 94.9 MB.
- RoPE + KV writeRoPE + KV write, 4,085 rows: rotate Q, K; copy K, V to cache: 56.5 MFLOP, 0.6 FLOP/B against a ridge of ~313, so memory-bound: reading 92.0 MB takes 11.5 µs, the math only 0.0 µs.
- FlashAttentionFlashAttention, 4,085 queries × all earlier keys, 64 heads: 137 GFLOP, 666 FLOP/B against a ridge of ~313, so compute-bound: the matrix cores need 54.7 µs, more than the 25.7 µs to read 205 MB.
- Output projectionOutput projection, [4,085 × 4,096] · [4,096 × 2,880]: 96.4 GFLOP, 1,196 FLOP/B against a ridge of ~313, so compute-bound: the matrix cores need 38.6 µs, more than the 10.1 µs to read 80.6 MB.
- RMSNormRMSNorm, 4,085 rows × 2,880: 47.1 MFLOP, 1.0 FLOP/B against a ridge of ~313, so memory-bound: reading 47.1 MB takes 5.9 µs, the math only 0.0 µs.
- Router + top-4Router + top-4, [4,085 × 2,880] · [2,880 × 128]: 3.0 GFLOP, 119 FLOP/B against a ridge of ~313, so memory-bound: reading 25.3 MB takes 3.2 µs, the math only 1.2 µs.
- Experts gate/up (grouped GEMM)Experts gate/up (grouped GEMM), 16,340 token-expert pairs, ~127.7 per expert, 128 experts hit: 542 GFLOP, 384 FLOP/B against a ridge of ~625, so memory-bound: reading 1.4 GB takes 176.3 µs, the math only 108.4 µs.
- Experts down (grouped GEMM)Experts down (grouped GEMM), 16,340 pairs, [2,880 → 2,880] per expert: 271 GFLOP, 360 FLOP/B against a ridge of ~625, so memory-bound: reading 752 MB takes 94.0 µs, the math only 54.2 µs.
- RMSNormRMSNorm, 64 rows × 2,880: 737 kFLOP, 1.0 FLOP/B against a ridge of ~313, so memory-bound: reading 743 kB takes 0.1 µs, the math only 0.0 µs.
- QKV projectionQKV projection, [64 × 2,880] · [2,880 × 5,120]: 1.9 GFLOP, 62 FLOP/B against a ridge of ~313, so memory-bound: reading 30.5 MB takes 3.8 µs, the math only 0.8 µs.
- RoPE + KV writeRoPE + KV write, 64 rows: rotate Q, K; copy K, V to cache: 885 kFLOP, 0.6 FLOP/B against a ridge of ~313, so memory-bound: reading 1.4 MB takes 0.2 µs, the math only 0.0 µs.
- Paged attentionPaged attention, 64 seqs × 1 query × 4,000 cached keys: 4.2 GFLOP, 8.0 FLOP/B against a ridge of ~313, so memory-bound: reading 525 MB takes 65.7 µs, the math only 1.7 µs.
- Output projectionOutput projection, [64 × 4,096] · [4,096 × 2,880]: 1.5 GFLOP, 62 FLOP/B against a ridge of ~313, so memory-bound: reading 24.5 MB takes 3.1 µs, the math only 0.6 µs.
- RMSNormRMSNorm, 64 rows × 2,880: 737 kFLOP, 1.0 FLOP/B against a ridge of ~313, so memory-bound: reading 743 kB takes 0.1 µs, the math only 0.0 µs.
- Router + top-4Router + top-4, [64 × 2,880] · [2,880 × 128]: 47.2 MFLOP, 42 FLOP/B against a ridge of ~313, so memory-bound: reading 1.1 MB takes 0.1 µs, the math only 0.0 µs.
- Experts gate/up (grouped GEMM)Experts gate/up (grouped GEMM), 256 token-expert pairs, ~2.0 per expert, 111 experts hit: 8.5 GFLOP, 8.6 FLOP/B against a ridge of ~625, so memory-bound: reading 985 MB takes 123.1 µs, the math only 1.7 µs. Most experts get one or two tokens, so their weights are read for almost no math.
- Experts down (grouped GEMM)Experts down (grouped GEMM), 256 pairs, [2,880 → 2,880] per expert: 4.2 GFLOP, 8.6 FLOP/B against a ridge of ~625, so memory-bound: reading 493 MB takes 61.6 µs, the math only 0.8 µs. Most experts get one or two tokens, so their weights are read for almost no math.
Assumptions behind every number on this page (all computed in the page’s _example.ts): BF16 activations, KV cache and attention/router weights; MXFP4 expert weights at 4.25 bits each, with expert math at the FP8 peak (the w4a8 path, not verified); MI355X spec peaks of 8 TB/s, 2.5 PFLOP/s BF16 and 5.0 PFLOP/s FP8 (approximate); uniform expert routing; FlashAttention with 128-row query blocks. moe_align_block_size (sorting tokens by expert) and the residual adds are left out: small next to the rest.
Worked example: one token through one layer
Follow one prompt token through layer 1 (full attention)
1 · In: a vector of 2,880 numbers
This token: 11.5 kFLOP (element-wise) · RMSNorm
Prefill, 4,085 prompt tokens
- Shape
- 4,085 rows × 2,880
- Work
- 47.1 MFLOP · 47.1 MB from HBM
- Intensity
- 1.0 FLOP/B vs ridge ~313
- Time ≥
- 5.9 µs, memory-bound
Decode, 64 sequences × 1 token
- Shape
- 64 rows × 2,880
- Work
- 737 kFLOP · 743 kB from HBM
- Intensity
- 1.0 FLOP/B vs ridge ~313
- Time ≥
- 0.1 µs, memory-bound
The token arrives as its hidden state: 2,880 BF16 values, 5,760 bytes. RMSNorm divides it by its root-mean-square and scales it: a few FLOPs per value, so about 1 FLOP per byte in both phases. Servers fuse it into a neighbour so it costs no separate trip to HBM.
2 · QKV projection: 2,880 → 5,120
This token: 29.5 MFLOP · QKV projection
Prefill, 4,085 prompt tokens
- Shape
- [4,085 × 2,880] · [2,880 × 5,120]
- Work
- 120 GFLOP · 94.9 MB from HBM
- Intensity
- 1,270 FLOP/B vs ridge ~313
- Time ≥
- 48.2 µs, compute-bound
Decode, 64 sequences × 1 token
- Shape
- [64 × 2,880] · [2,880 × 5,120]
- Work
- 1.9 GFLOP · 30.5 MB from HBM
- Intensity
- 62 FLOP/B vs ridge ~313
- Time ≥
- 3.8 µs, memory-bound
One GEMM produces the query (64 heads × 64 = 4,096 values), key (8 × 64 = 512) and value (512) for the token. The 29.5 MB weight matrix is read once per step: in prefill 4,085 tokens share that read, in decode only 64.
3 · RoPE and the KV write
This token: 13.8 kFLOP (element-wise) · RoPE + KV write
Prefill, 4,085 prompt tokens
- Shape
- 4,085 rows: rotate Q, K; copy K, V to cache
- Work
- 56.5 MFLOP · 92.0 MB from HBM
- Intensity
- 0.6 FLOP/B vs ridge ~313
- Time ≥
- 11.5 µs, memory-bound
Decode, 64 sequences × 1 token
- Shape
- 64 rows: rotate Q, K; copy K, V to cache
- Work
- 885 kFLOP · 1.4 MB from HBM
- Intensity
- 0.6 FLOP/B vs ridge ~313
- Time ≥
- 0.2 µs, memory-bound
RoPE rotates q and k by the token’s position. Then reshape_and_cache copies k and v (1,024 values, 2 KiB) into the token’s slot in the paged KV cache (see the kernel below). Pure data movement: under 1 FLOP per byte.
4 · Attention: look back at earlier tokens
This token: 33.5 MFLOP · FlashAttention
Prefill, 4,085 prompt tokens
- Shape
- 4,085 queries × all earlier keys, 64 heads
- Work
- 137 GFLOP · 205 MB from HBM
- Intensity
- 666 FLOP/B vs ridge ~313
- Time ≥
- 54.7 µs, compute-bound
Decode, 64 sequences × 1 token
- Shape
- 64 seqs × 1 query × 4,000 cached keys
- Work
- 4.2 GFLOP · 525 MB from HBM
- Intensity
- 8.0 FLOP/B vs ridge ~313
- Time ≥
- 65.7 µs, memory-bound
In prefill, FlashAttention scores every token against all earlier ones in tiles; each K/V block it reads serves 128 query rows and 8 query heads, so it is compute-bound. In decode each sequence brings one query and reads its whole cache (4,000 tokens × 2 KiB): about 8 FLOP/B, memory-bound, and batching cannot help.
5 · Sliding-window layers are cheaper
This token: 2.1 MFLOP · FlashAttention
Prefill, 4,085 prompt tokens
- Shape
- 4,085 queries × ≤ 128 keys, 64 heads
- Work
- 8.4 GFLOP · 83.4 MB from HBM
- Intensity
- 101 FLOP/B vs ridge ~313
- Time ≥
- 10.4 µs, memory-bound
Decode, 64 sequences × 1 token
- Shape
- 64 seqs × 1 query × 128 cached keys
- Work
- 134 MFLOP · 17.8 MB from HBM
- Intensity
- 7.5 FLOP/B vs ridge ~313
- Time ≥
- 2.2 µs, memory-bound
Half of gpt-oss’s layers only look back 128 tokens. There, attention costs the same per token however long the context is: in prefill it drops from compute-bound to a small memory-bound kernel; in decode the cache read shrinks from 4,000 to 128 tokens.
6 · Output projection: 4,096 → 2,880
This token: 23.6 MFLOP · Output projection
Prefill, 4,085 prompt tokens
- Shape
- [4,085 × 4,096] · [4,096 × 2,880]
- Work
- 96.4 GFLOP · 80.6 MB from HBM
- Intensity
- 1,196 FLOP/B vs ridge ~313
- Time ≥
- 38.6 µs, compute-bound
Decode, 64 sequences × 1 token
- Shape
- [64 × 4,096] · [4,096 × 2,880]
- Work
- 1.5 GFLOP · 24.5 MB from HBM
- Intensity
- 62 FLOP/B vs ridge ~313
- Time ≥
- 3.1 µs, memory-bound
The 64 head outputs (4,096 values) are mapped back to 2,880 and added to the residual stream. Same pattern as step 2: compute-bound with thousands of rows, memory-bound with 64.
7 · Router: pick 4 of 128 experts
This token: 737 kFLOP · Router + top-4
Prefill, 4,085 prompt tokens
- Shape
- [4,085 × 2,880] · [2,880 × 128]
- Work
- 3.0 GFLOP · 25.3 MB from HBM
- Intensity
- 119 FLOP/B vs ridge ~313
- Time ≥
- 3.2 µs, memory-bound
Decode, 64 sequences × 1 token
- Shape
- [64 × 2,880] · [2,880 × 128]
- Work
- 47.2 MFLOP · 1.1 MB from HBM
- Intensity
- 42 FLOP/B vs ridge ~313
- Time ≥
- 0.1 µs, memory-bound
A small GEMM scores the token against 128 experts; top-4 keeps the best four and their weights. Cheap in both phases. What matters is what it decides: in prefill each expert gets about 128 tokens; in decode the 256 picks of 64 tokens land on about 111 experts, ~2 each (uniform routing assumed).
8 · Experts gate/up: 2,880 → 5,760, then SwiGLU
This token: 133 MFLOP · Experts gate/up (grouped GEMM)
Prefill, 4,085 prompt tokens
- Shape
- 16,340 token-expert pairs, ~127.7 per expert, 128 experts hit
- Work
- 542 GFLOP · 1.4 GB from HBM
- Intensity
- 384 FLOP/B vs ridge ~625
- Time ≥
- 176.3 µs, memory-bound
Decode, 64 sequences × 1 token
- Shape
- 256 token-expert pairs, ~2.0 per expert, 111 experts hit
- Work
- 8.5 GFLOP · 985 MB from HBM
- Intensity
- 8.6 FLOP/B vs ridge ~625
- Time ≥
- 123.1 µs, memory-bound
Each of the token’s 4 experts multiplies it by its own 2,880 × 5,760 matrix (MXFP4, 8.8 MB per expert), then SwiGLU combines the two halves to 2,880. A grouped GEMM runs all experts in one launch. This is the largest weight read of the layer, and in decode it is read for ~2 tokens per expert.
9 · Experts down: 2,880 → 2,880, combine
This token: 66.4 MFLOP · Experts down (grouped GEMM)
Prefill, 4,085 prompt tokens
- Shape
- 16,340 pairs, [2,880 → 2,880] per expert
- Work
- 271 GFLOP · 752 MB from HBM
- Intensity
- 360 FLOP/B vs ridge ~625
- Time ≥
- 94.0 µs, memory-bound
Decode, 64 sequences × 1 token
- Shape
- 256 pairs, [2,880 → 2,880] per expert
- Work
- 4.2 GFLOP · 493 MB from HBM
- Intensity
- 8.6 FLOP/B vs ridge ~625
- Time ≥
- 61.6 µs, memory-bound
Each expert maps back to 2,880; the four results are summed with the router’s weights and added to the residual. The token leaves the layer as 2,880 numbers again and enters layer 2.
10 · The whole step: 36 layers
- Prefill step, lower bound
- 15.12 ms
- 4,085 tokens, 18 full + 18 sliding layers + LM head
- Decode step, lower bound
- 8.29 ms
- 64 sequences, 4,000 cached tokens each
| prefill | decode | |
|---|---|---|
| RMSNorm | 1.4 | 0.0 |
| QKV projection | 11.5 | 1.7 |
| RoPE + KV write | 2.7 | 0.1 |
| FlashAttention | 7.8 | 14.8 |
| Output projection | 9.2 | 1.3 |
| RMSNorm | 1.4 | 0.0 |
| Router + top-4 | 0.8 | 0.1 |
| Experts gate/up | 42.0 | 53.5 |
| Experts down | 22.4 | 26.8 |
Summed over all 36 layers, the roofline says a 4,085-token prefill needs at least 15.12 ms on one MI355X. Real prefills take several times longer: real kernels do not reach peak, and launches and scheduling add time. In both phases the experts dominate; in decode, attention’s share grows with context length.
“This token” = the FLOPs spent on one token in prefill. Both columns are whole-kernel numbers on one MI355X, lower bounds.
Prefill vs decode, side by side
| Property | Prefill | Decode |
|---|---|---|
| Rows per GEMM | thousands | batch size (tens) |
| Arithmetic intensity (computed above) | QKV 1,270, experts 384 FLOP/B (T = 4,085) | QKV 62, experts 8.6 FLOP/B (B = 64) |
| Bound by | matrix-core FLOPs | HBM bandwidth (weights + KV cache) |
| GEMM tiles / workgroups | hundreds to thousands; fills 256 CUs | few; needs split-K, small tiles or GEMV |
| Attention kernel | FlashAttention: Q block × streamed K/V, causal | paged decode attention: one query, block-table lookups, split-KV |
| MoE | grouped GEMM with well-filled tiles | grouped GEMM mostly padding; each expert’s weights read for 1–3 tokens |
| Time per step | tens of ms for a few thousand tokens | ~ms; kernel launch overhead becomes visible |
| Launch strategy | normal launches | often graph capture (CUDA/HIP graphs): the whole step’s kernel sequence recorded once, replayed with one launch |
| LM head [M × 2,880] · [2,880 × 201,088] | only for the last prompt token | every sequence, every step: 1.2 GB of weights read each time |
| Precision effect | FP8/FP4 matrix instructions raise peak FLOPs | smaller weights cut bytes read: near-proportional speed-up |
Move the slider to see where one GEMM crosses the ridge. Dense layers need a few hundred rows; the MoE experts need far more, because the rows are split over 128 experts:
Bytes = weights once + input and output rows (BF16). MoE: tokens spread over 128 experts, 4 each, uniform routing assumed; only experts that get a token are read; expert math at the FP8 peak (w4a8 assumed). Peaks: MI355X spec, 8 TB/s, 2.5 PFLOP/s BF16, 5.0 PFLOP/s FP8 (approximate).
Fusion
Element-wise steps (norms, RoPE, activation, residual add, quantizing activations to FP8) are memory-bound: each reads and writes the whole hidden state for a handful of FLOPs. Servers fuse them into neighbours: RMSNorm with the FP8 quantize, SiLU-and-multiply into the gate/up GEMM epilogue, RoPE with the KV write. In decode, where every step is short, fusion also cuts the number of kernel launches. Graph capture removes most of the remaining per-launch CPU cost.
Writing the KV cache: a real kernel
After the K and V projections, each token’s K and V rows must land in its slot of the paged KV cache. vLLM does this in reshape_and_cache_flash_kernel (csrc/cache_kernels.cu). Its key idea is simple:
- The scheduler (on the CPU) has already decided where each token goes and passes a
slot_mappingarray: one slot number per token. . A value of −1 marks a padded token to skip. - The kernel runs one workgroup per token. Its threads copy that token’s K and V rows (all KV heads) into
key_cache[block][offset]andvalue_cache[block][offset], 16 bytes at a time.
A simplified version (NHD layout, BF16, no FP8 scaling), written for this page from the vLLM kernel and compiled for both vendors:
// Simplified from vLLM reshape_and_cache_flash_kernel. Compiles with nvcc and hipcc.
struct alignas(16) vec16 { unsigned int w[4]; }; // 8 BF16 values
extern "C" __global__ void kv_write(const vec16* key, const vec16* value,
vec16* key_cache, vec16* value_cache,
const long long* slot_mapping, int block_size, int row_vecs) {
long long token = blockIdx.x; // one workgroup per token
long long slot = slot_mapping[token];
if (slot < 0) return; // padded token
long long block = slot / block_size, offset = slot % block_size;
long long dst = (block * block_size + offset) * row_vecs;
for (int i = threadIdx.x; i < row_vecs; i += blockDim.x) {
key_cache[dst + i] = key[token * row_vecs + i];
value_cache[dst + i] = value[token * row_vecs + i];
}
}
Example PTX (nvcc -arch=sm_90 -ptx, illustrative, trimmed):
mov.u32 %r6, %ctaid.x; // token = blockIdx.x
mul.wide.u32 %rd15, %r6, 8;
add.s64 %rd16, %rd14, %rd15;
ld.global.nc.u64 %rd2, [%rd16]; // slot = slot_mapping[token]
setp.lt.s64 %p1, %rd2, 0;
@%p1 bra $L__BB0_4; // padded: skip
...
mul.lo.s64 %rd8, %rd2, %rd17; // dst = slot * row_vecs (compiler folded block*bs+offset back to slot)
$L__BB0_3: // copy loop
ld.global.nc.v4.u32 {%r7, %r8, %r9, %r10}, [%rd22]; // 16 B of K
st.global.v4.u32 [%rd24], {%r7, %r8, %r9, %r10};
ld.global.nc.v4.u32 {%r15, %r16, %r17, %r18}, [%rd25]; // 16 B of V
st.global.v4.u32 [%rd26], {%r15, %r16, %r17, %r18};
add.s32 %r23, %r23, %r2; // i += blockDim.x
setp.lt.s32 %p3, %r23, %r5;
@%p3 bra $L__BB0_3;
Example gfx950 assembly (example ROCm 7.0.0, --save-temps, illustrative, trimmed):
s_lshl_b64 s[6:7], s[2:3], 3 ; token * 8 bytes
s_add_u32 s4, s4, s6
s_addc_u32 s5, s5, s7
s_load_dwordx2 s[12:13], s[4:5], 0x0 ; slot = slot_mapping[token] -> SCALAR load
s_waitcnt lgkmcnt(0)
v_cmp_lt_i64_e64 s[4:5], s[12:13], 0 ; slot < 0 ?
s_and_b64 vcc, exec, s[4:5]
s_cbranch_vccnz .LBB0_4 ; padded: whole wave exits
...
.LBB0_3: ; copy loop, per lane
global_load_dwordx4 v[2:5], v[2:3], off ; 16 B of K
global_load_dwordx4 v[6:9], v[6:7], off ; 16 B of V
...
s_waitcnt vmcnt(1)
global_store_dwordx4 v[12:13], v[2:5], off
s_waitcnt vmcnt(1)
global_store_dwordx4 v[10:11], v[6:9], off
s_andn2_b64 exec, exec, s[2:3] ; lanes that finished switch off
s_cbranch_execnz .LBB0_3
What to notice:
- Uniform values go scalar on AMD.
slotis the same for every thread in the workgroup, so the compiler loads it once withs_load_dwordx2into SGPRs and branches with the scalar unit. NVIDIA loads it per thread (ld.global.nc.u64), though hardware merges the identical addresses. - It is a pure copy. No math beyond address arithmetic: 16-byte loads and stores, . It is memory-bound and cheap: for gpt-oss one token is 8 KV heads × 64 × 2 B = 1 KB of K plus 1 KB of V per layer. Prefill of 4,000 tokens writes about 8 MB per layer, about 0.3 GB for all 36 layers (about 72 KB per token).
- The layout is a contract. The attention kernel that reads the cache must agree on block size and on the order
[block][offset][head][dim](NHD) vs[block][head][offset][dim](HND). In disaggregated serving the prefill side also ships these blocks to a separate decode side, which may expect a different layout.
Seeing real kernels
You do not need to read source to learn what a server runs. Profilers list every kernel launch with its name, grid, registers, LDS and duration.
AMD, rocprofv3 (rocprofiler-sdk):
# Trace every kernel dispatch and add per-kernel statistics (rocpd database by default).
rocprofv3 --kernel-trace --stats -d ./prof -- python3 my_bench.py
# Convert to CSV when needed.
rocpd convert -i ./prof/<...>_results.db --output-format csv
The kernel trace has one row per dispatch with Kernel_Name, Start_Timestamp, End_Timestamp, LDS_Block_Size, Scratch_Size, VGPR_Count, Accum_VGPR_Count, SGPR_Count, workgroup and grid sizes. Sorting by total time shows where a step goes. For hardware counters (cache hit rates, MFMA busy) use --pmc or rocprofiler-compute (formerly Omniperf).
NVIDIA, Nsight Systems and Nsight Compute (CUDA CLI):
nsys profile -t cuda,nvtx --stats=true python3 my_bench.py # timeline + per-kernel summary
ncu --set full -k regex:fused_moe -c 1 python3 my_bench.py # one kernel, all sections incl. roofline
Nsight Systems answers “which kernels, how long, in what order”. Nsight Compute answers “why is this kernel slow”: occupancy, memory throughput, a roofline chart, and per-instruction stalls.
Inside the framework. torch.profiler gives the same per-kernel list from Python and maps kernels back to model layers. vLLM logs which attention backend it picked at start-up. → FlashAttention: vLLM backends
How to read a trace of one step:
| You see | It means |
|---|---|
| A few GEMM kernels take most of the time | prefill-like, compute-bound; check matrix-core utilization |
| Hundreds of short kernels with gaps between them | decode-like; launch overhead; graph capture missing or not covering this batch size |
fused_moe_kernel dominating |
MoE layers; in decode, mostly padding and weight streaming |
| Attention dominating | long contexts; check backend and KV dtype |
Names starting Cijk_ |
Tensile GEMM kernels from rocBLAS/hipBLASLt |
Common misconceptions
- “Decode is just prefill with one token.” The math is the same; the kernels are not. Different attention kernel, different GEMM tiles, graph capture, and the opposite bottleneck.
- “Batching decode fixes everything.” It raises intensity for dense GEMMs, but not for attention (each sequence has its own KV) and only weakly for MoE (tokens spread over 128 experts).
- “The GPU is idle between kernels, so it is free.” In decode those gaps are part of the step time, which is why launch overhead and graph capture matter.
How this shows up in operations
- Prefill and decode want different hardware. Prefill saturates GPU matrix cores; decode is bandwidth-bound, which is why some systems run the two phases on different hardware.
- Prefill Pods are sized by HBM, then by FLOPs. Weights plus KV cache must fit; after that, prompt throughput follows matrix-core speed.
- The KV write is the hand-off point. Prefill writes KV with
reshape_and_cache, then the connector ships it over RDMA. A layout or block-size mismatch between the two sides is a correctness bug, not a performance one.
Further reading
- vLLM
csrc/cache_kernels.cu:reshape_and_cache_flash_kernel,slot_mapping - vLLM
fused_moe.py: grouped MoE GEMM - PagedAttention / vLLM paper (arXiv 2309.06180)
- rocprofv3: kernel trace · rocprofiler-compute
- Nsight Systems user guide · Nsight Compute CLI
- gpt-oss-120b config
- AITER: AMD-tuned attention, MoE and GEMM kernels used by vLLM on ROCm