Learn · Kernels & silicon
GPU memory hierarchy and the roofline
A GPU can do arithmetic far faster than it can fetch the numbers to do it on. An MI355X can perform about 600 FP8 operations in the time it takes to read one byte from HBM. Almost every kernel optimization is about reusing data that is already close to the math units. This page lists the levels of memory, how big and how far each is, and then gives the one model that predicts whether a kernel is limited by math or by memory: the roofline.
The idea first
A kitchen with one road to the warehouse. Picture a kitchen with thousands of fast cooks (the GPU’s math units) and one delivery road to a huge warehouse far away (HBM, the GPU’s main memory). The cooks can chop faster than the road can bring ingredients. Shelves next to the stove (the caches, LDS and registers) are small but instant. Every speed question on a GPU reduces to: for each ingredient that comes down the road, how much cooking happens?
| Piece | Kitchen version | The question it answers |
|---|---|---|
| Registers | what each cook holds in their hands | “Is the number I need already here, right now?” |
| LDS / shared memory | the team’s shared table; the team decides what goes on it | “What should my team fetch once and share?” |
| L1, L2, Infinity Cache | shelves the kitchen restocks by itself with what was used recently | “Did someone fetch this recently?” |
| HBM | the warehouse: holds everything (288 GB), far away | “How fast can the road deliver?” (~8 TB/s) |
| Arithmetic intensity | cooking steps per kilogram delivered (FLOPs per byte) | “Is this recipe heavy on cooking or on ingredients?” |
| Ridge point | the recipe where road and cooks are exactly matched | “Which one will run out first?” |
- A salad: little cooking per ingredientAdding two arrays (
y = a·x + y) is one quick chop per ingredient: about 0.17 FLOP per byte. The road is the limit; faster cooks would change nothing. This is memory-bound, and so is LLM decode. - A stew: lots of cooking per ingredientA big matrix multiply uses each number thousands of times once it is on the shelf. The cooks are the limit. This is compute-bound, and so is LLM prefill.
- The break-even recipeDivide cook speed by road speed: 2,500 × 10¹² ÷ 8 × 10¹² ≈ 310 FLOPs per byte. Recipes below that wait on the road; recipes above it wait on the cooks. That number is the ridge point.
Where the analogy breaks. The road delivers in fixed crates (128-byte cache lines), so asking for one scattered gram wastes a whole crate. The cooks never stand idle by choice: while one group waits for a delivery, another group cooks (latency hiding). And the kitchen restocks the automatic shelves on its own rules; only the team table (LDS) is under the kernel’s control.
The levels on an MI355X
- L1 hit. The data was used recently by this CU. Shortest trip after registers and LDS.
- L1 miss, L2 hit. Another workgroup on the same XCD read this line recently.
- L2 miss, Infinity Cache hit. Shared by all 8 XCDs, so a line another XCD fetched is caught here.
- Miss everywhere: the line comes from HBM. This is the trip every kernel tries to make rare.
- LDS read. No cache lookup at all: the kernel copied the data here earlier and knows it is present.
| Level | MI355X (CDNA4) | H100 SXM (Hopper) | B200 (Blackwell) | Managed by | Visible to |
|---|---|---|---|---|---|
| Registers | 512 KB VGPR per CU (128 MB per GPU) | 256 KB per SM (33 MB per GPU) | 256 KB per SM | compiler | one thread (lane) |
| LDS / shared memory | 160 KB per CU | up to 228 KB per SM, carved out of a 256 KB L1/shared array | same as H100 (228 of 256 KB) | the kernel, explicitly | one workgroup / block |
| L1 data cache | 32 KB per CU (vector) + 16 KB scalar per 2 CUs | the rest of the 256 KB array | same | hardware | one CU / SM |
| L2 | 4 MB per XCD, 32 MB total | 50 MB | 126 MB (NVIDIA tuning guide, GB200) | hardware | one XCD (AMD) / whole GPU |
| Last-level cache | 256 MB Infinity Cache | — | — | hardware | whole GPU |
| HBM | 288 GB HBM3E, ~8 TB/s | 80 GB HBM3, ~3.35 TB/s | 180 GB HBM3E, ~8 TB/s | allocator (hipMalloc, cudaMalloc) |
whole GPU, and peers via links |
Sources: MI355X fields as rocminfo reports them (per the public specs), ROCm GPU hardware specs, CUDA compute-capability tables, Hopper and Blackwell tuning guides, the Hopper architecture blog, vendor product pages. HBM bandwidths are spec-sheet peaks and therefore approximate; B200 bandwidth is NVIDIA’s DGX B200 figure (64 TB/s for 8 GPUs) divided by 8.
Three things stand out:
- The register file is the biggest fast memory on the chip. 128 MB of VGPRs on an MI355X is four times its L2. Keeping a tile in registers is the most effective reuse there is.
- LDS is software-managed. Nothing is cached in it automatically. A kernel copies data in with
ds_write(orglobal_load_ldson gfx950), synchronizes with a barrier, and reads it withds_read. That gives exact control, which is why fast kernels use it. - AMD’s L2 is per die. The 32 CUs of an XCD share 4 MB. Data used by CUs on different XCDs is fetched into each XCD’s L2 separately; the 256 MB Infinity Cache sits behind all of them and catches much of that.
Latency grows at each step outward: registers are read every cycle, LDS in tens of cycles, HBM in hundreds of nanoseconds. Vendors do not publish exact latencies; treat those as orders of magnitude. The execution model page shows how resident waves hide them.
Worked example: one wavefront’s load
Follow one instruction, global_load_dwordx4, issued by one wave of 64 lanes on an MI355X, and see how many bytes each level must move. All numbers below are computed from the access pattern.
One wave loads data, then reads it from LDS
1 · Coalesced: lane i reads bytes 16i to 16i + 15
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| lanes 0–15 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| lanes 16–31 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 |
| lanes 32–47 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 5 |
| lanes 48–63 | 6 | 6 | 6 | 6 | 6 | 6 | 6 | 6 | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 |
Each lane asks for 16 bytes, neighbours ask for neighbouring bytes. The 64 requests merge (coalesce) into 8 cache lines: 1,024 bytes fetched, 1,024 bytes used. 100 % efficiency, one instruction.
2 · Strided: lane i reads x[32 · i]
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| lanes 0–15 | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 |
| lanes 16–31 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 |
| lanes 32–47 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 |
| lanes 48–63 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 |
Now each lane reads one float from a column of a row-major matrix 32 floats wide: addresses 128 bytes apart. Every lane lands in its own line: 64 lines, 8,192 bytes fetched to use 256. 3.1 % efficiency. Same instruction count, 32× the traffic. Fix: change the data layout, or load rows and transpose in LDS.
3 · The trip out and back
- Lines requested
- 8
- × 128 B, coalesced case
- Bytes in flight to fill HBM
- 4 MB
- 8 TB/s × 500 ns (assumed)
- Per CU
- 15.259 KiB
- ÷ 256 CUs
- Waves per CU
- ~15
- each with one 1 KiB load outstanding
The L1 misses, so the request goes to this XCD’s L2, then the Infinity Cache, then HBM (diagram above). The whole GPU moves 1 KiB in about 128 ps, but each request waits hundreds of nanoseconds. To keep HBM busy, about 15 waves per CU must have a load like this outstanding at once (Little’s law; see latency hiding).
4 · Into LDS, then a column read: 64-way bank conflict
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| lanes 0–15 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| lanes 16–31 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| lanes 32–47 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| lanes 48–63 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
Lane i reads row i, column 0. Bank = (address ÷ 4) mod 64.
The kernel writes the data to LDS (ds_write_b128) so the whole workgroup can reuse it. Then each lane reads column 0 of a 64 × 64 float tile: lane i reads row i. Rows are 256 bytes apart, a multiple of 64 banks × 4 B, so all 64 lanes hit bank 0. LDS serves one address per bank per pass: 64 passes instead of 1.
5 · Pad each row by one float: no conflict
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| lanes 0–15 | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 |
| lanes 16–31 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 |
| lanes 32–47 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 |
| lanes 48–63 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 |
Allocate the tile 65 floats wide and use only 64. Row i now starts at bank i, so the column spreads over all 64 banks: worst bank load 1, one pass. Kernels either pad like this or swizzle (XOR the column index with the row). gfx950 also has transposing LDS reads (ds_read_b64_tr_b16) that return matrix fragments in the layout MFMA wants. → PTX and AMDGCN
6 · Registers: where the reuse pays off
The fragment now sits in VGPRs, read every cycle by the vector and matrix units at no extra cost. Every FLOP done on it here is a FLOP that did not need a new byte from HBM. Counting those FLOPs per byte is exactly the roofline, next.
Cache line 128 B (rocminfo), 64 LDS banks of 4 B (CDNA4 ISA guide). Latency is assumed, not measured.
The roofline
Arithmetic intensity ():
A kernel’s best possible speed is the smaller of two ceilings:
- memory ceiling:
- compute ceiling: peak FLOP/s
They meet at the ridge point, . Below it the kernel is memory-bound: a faster ALU would not help. Above it, it is compute-bound: more bandwidth would not help.
Ridge points
| GPU | Precision (dense, no sparsity) | Peak (spec, approx.) | HBM (approx.) | Ridge point |
|---|---|---|---|---|
| MI355X | FP16 / BF16 matrix | 2.5 PFLOP/s | 8 TB/s | ~310 FLOP/B |
| MI355X | FP8 matrix | 5.0 PFLOP/s | 8 TB/s | ~625 FLOP/B |
| MI355X | MXFP4 / MXFP6 matrix | 10.1 PFLOP/s | 8 TB/s | ~1,260 FLOP/B |
| H100 SXM | FP16 / BF16 Tensor | ~990 TFLOP/s | 3.35 TB/s | ~295 FLOP/B |
| H100 SXM | FP8 Tensor | ~1,980 TFLOP/s | 3.35 TB/s | ~590 FLOP/B |
| B200 | FP8 Tensor | ~4.5 PFLOP/s | ~8 TB/s | ~560 FLOP/B |
AMD lists MI355X dense numbers directly. NVIDIA’s H100 page lists FP8 as 3,958 TFLOP/s with sparsity; dense is half. The B200 FP8 value is NVIDIA’s DGX B200 72 PFLOP/s (8 GPUs, sparse) ÷ 8 ÷ 2, so it is derived, not quoted.
The lesson: on every current datacenter GPU a kernel needs a few hundred FLOPs per byte from HBM before the matrix cores become the limit. Lower precision moves the ridge up: FP8 doubles peak FLOPs, so the same kernel must reuse data twice as much to stay compute-bound.
Try it
Pick a kernel and a GPU. The red dot sits at the kernel’s intensity; its height is the best speed the roofline allows. The Q/K/V presets use gpt-oss-120b’s projection shape ( in, out, published config).
What to notice:
- Element-wise add sits at the far left (about 0.17 FLOP/B in BF16, 1 FLOP per 6 bytes). Activations, residual adds and norms are always memory-bound; the fix is to fuse them into a neighbouring kernel so their data skips a trip to HBM.
- Decode, 1 sequence: , so each weight is used once. Intensity . Batch 64 moves it about 64× right, still below the ridge.
- Prefill and the square GEMM sit far right: compute-bound, if the kernel achieves near-minimum traffic. That is what tiling is for.
- Switch FP8 → MXFP4 on MI355X: the dot moves right (fewer bytes), but the ridge moves right too (more FLOP/s).
Apply it to an LLM
A linear layer with weight matrix does 2 FLOPs per weight per token. If tokens go through it together and each weight is stored in bytes, then (activations are small next to weights here).
gpt-oss-120b stores its expert weights in MXFP4: 4-bit values plus one 8-bit scale per 32 values, about 0.53 bytes per weight (format arithmetic; published config says quant_method: mxfp4). The comparison point is the FP8 ridge, about 625, assuming 8-bit activations (a “w4a8” checkpoint; see tokens and transformers).
| Phase (dense weights: attention, shared layers) | Tokens per weight read () | Against the ~625 ridge | |
|---|---|---|---|
| Decode, 1 sequence | 1 | ~4 | memory-bound, ~150× below |
| Decode, batch 64 | 64 | ~240 | still memory-bound |
| Prefill, one 4,000-token prompt | 4,000 | ~15,000 | compute-bound |
| Prefill, same prompt, MoE expert weights (4 of 128 experts per token) | ~125 | ~470 | near the ridge, below it |
Mixture-of-experts makes decode worse. gpt-oss-120b has 128 experts and routes each token to 4 (published config). A decode batch of 64 tokens makes 256 expert assignments over 128 experts: about 2 tokens per expert weight read, not 64. A 4,000-token prefill gives about 125 tokens per expert. → LLM kernels
Attention in decode cannot be fixed by batching at all. Each sequence reads its own KV cache. For gpt-oss (64 query heads sharing 8 KV heads of size 64, 16-bit KV), each cached token costs 2,048 bytes of K and V per layer and yields about 16,000 FLOPs ( and for 64 heads): FLOP/B, whatever the batch size.
A rough upper bound: one decode step of gpt-oss-120b reads its ~5.1 B active parameters (published figure) at ~0.53 B each, about 2.7 GB. At 8 TB/s that is ~0.34 ms, so no more than ~3,000 steps per second for one sequence on one MI355X, before counting attention, KV reads or any overhead. (Order-of-magnitude estimate; attention weights are not MXFP4 and real kernels do not hit peak bandwidth.)
So: prefill wants FLOPs, decode wants bandwidth. That is the root of disaggregated inference: running the two phases on hardware suited to each.
Common misconceptions
- “More HBM capacity makes it faster.” Capacity decides what fits (weights + KV cache). Speed of decode follows bandwidth, not capacity.
- “Caches will handle reuse.” L2 and Infinity Cache help, but LLM weights (tens of GB) never fit. Reuse has to be designed into the kernel through tiling and batching.
- “Peak TFLOPS predicts speed.” Only for kernels above the ridge. Most of a decode step sits far below it.
- “FP4 makes everything 4× faster.” It cuts bytes per weight 4× versus BF16, which helps memory-bound decode almost 4×. Compute-bound prefill gains only where FP4 matrix instructions are used end to end.
How this shows up in operations
- HBM used is not HBM busy. A serving Pod holding most of an MI355X’s HBM says nothing about bandwidth use. Memory-bandwidth utilization comes from profiler counters, not
amd-smimemory usage. → GPU metrics - Partition modes change the cache picture. Splitting an MI355X into smaller compute partitions gives each partition fewer XCDs and their L2s. → GPU partitioning
- HBM errors are memory-hierarchy events. ECC errors and retired pages are reported per HBM stack/page. → GPU failures
Further reading
- ROCm GPU hardware specifications: LDS, L1, L2, Infinity Cache, VGPR file per AMD GPU
- AMD MI300 series microarchitecture: XCD, L2, Infinity Cache layout (same structure as MI355X)
- AMD CDNA4 ISA reference guide (PDF): LDS banks and memory instructions
- AMD MI355X specifications · NVIDIA H100 · NVIDIA DGX B200
- CUDA compute capabilities: memory per SM · shared-memory carve-outs
- Hopper tuning guide · Blackwell tuning guide: L2 sizes, shared memory
- Roofline model (Wikipedia)
- gpt-oss-120b config: experts, heads, MXFP4