Learn · GPU fundamentals
What a GPU is
A CPU has a few dozen large cores, each fast at one stream of branchy code. A GPU has thousands of small lanes that run the same instruction on different numbers at the same time, next to memory that is about ten times faster than server RAM. LLM math is mostly large matrix multiplies, which split into millions of identical small jobs. That is the whole reason LLM prompt processing runs on GPUs.
This page is the physical picture: what is inside the package, what the numbers on a spec sheet mean, and how the GPU sits in a server. Programming details are in threads and wavefronts and memory and roofline.
The idea first
A CPU is a few master chefs; a GPU is a huge kitchen of line cooks. A master chef can cook any dish, change plans mid-recipe and work alone. A line cook does one simple step, and a whole row of them does the same step at the same moment: 64 cooks all chop on the same beat. To make 10,000 identical salads, the line wins by far. A matrix multiply is that kind of job: millions of the same multiply-and-add on different numbers.
Every part of the chip answers one question:
| Part (MI355X name) | In the kitchen | The question it answers |
|---|---|---|
| Lane | one cook’s hands | who does one multiply-add on one number? |
| SIMD (16 lanes) and wavefront (64 threads) | a row of cooks on one beat | which threads must run the same instruction together? |
| Matrix core | a food processor next to the row | what does a whole small matrix multiply in one step? |
| CU (compute unit) | one kitchen station with its own counter (LDS) | where does one group of threads (a workgroup) run and share scratch space? |
| XCD (compute die) | one kitchen floor | how many stations are built on one piece of silicon? |
| IOD + HBM | the pantry and the corridors | where do the ingredients (weights, KV cache) live and how fast do they arrive? |
Where the analogy breaks. Cooks in a row that need to do different things (an if that goes both ways) take turns, while the others wait. A station cannot borrow another station’s counter. And the kitchen is usually limited by the pantry, not the cooks: for LLM decode the matrix cores wait on HBM most of the time (worked out below).
Zoom in: package to lane
One MI355X, from the package down to one lane
1 · The package: 8 compute dies on 2 I/O dies, 8 memory stacks
36 GBHBM3E
36 GBHBM3E
36 GBHBM3E
36 GB
36 GBHBM3E
36 GBHBM3E
36 GBHBM3E
36 GB
Ten dies share one package. The XCDs (accelerator complex dies, TSMC N3P) do the math; they sit stacked on top of two IODs (I/O dies, TSMC N6) that hold the 256 MB Infinity Cache, the memory controllers and the links to the host and the other GPUs. Beside them, 8 HBM3E stacks of 36 GB give 288 GB. We zoom into XCD 0.
2 · One XCD: 36 compute units built, 32 enabled
CU the one we zoom into · × disabled spare (positions illustrative)
Each XCD is built with 36 CUs; AMD enables 32 so that a die with a few defects is still usable. 8 × 32 = 256 CUs per GPU (288 built). The CUs of one XCD share a 4 MB L2 cache, 32 MB across the GPU.
3 · One CU: 4 SIMDs, 4 matrix cores, 160 KB of LDS
A CU is the closest thing to a CPU core: it has its own instruction scheduler, registers and caches. It holds 4 SIMD units for ordinary math (activations, softmax, normalisation) and a matrix core beside each one for small matrix multiplies. The LDS (local data share) is 160 KB of fast scratch memory that the threads of one workgroup share. One CU can keep up to 32 wavefronts resident and switches between them to hide memory waits.
4 · One SIMD: 16 lanes run a 64-thread wavefront in 4 passes
Columns: the 16 lanes of one SIMD. Numbers: the 64 threads of one wavefront, all running the same instruction, fed through in 4 passes.
A wavefront is 64 threads that execute the same instruction. The SIMD has 16 lanes, so one instruction for the whole wavefront goes through in 4 passes. A lane is what spec sheets call a “stream processor”: 256 × 4 × 16 = 16,384. It is not a core: it has no program counter of its own.
5 · Add it up
- CUs
- 256
- 8 XCDs × 32
- Lanes
- 16,384
- 1,024 SIMDs × 16
- Matrix cores
- 1,024
- 2,048 FP8 FLOPs per clock each
- Threads resident
- 524,288
- 32 waves × 64 × 256 CUs
- Dense FP8 peak
- 5.03 PF
- 8192 FLOPs/clock/CU × 256 × 2400 MHz
- HBM
- 288 GB
- 8,192 GB/s = 8192 bits × 8 Gb/s
Almost all LLM FLOPs come from the matrix cores; the lanes do the rest. The last tile is the pantry: 8,192 GB/s sounds huge, but at 5.0 PFLOPS the matrix cores could do about 614 FP8 operations for every byte HBM delivers.
6 · The same zoom on NVIDIA Blackwell Ultra
| Level | AMD MI355X | NVIDIA Blackwell Ultra (B300, GB300) |
|---|---|---|
| Package | 8 XCDs + 2 IODs | 2 reticle-sized dies joined by NV-HBI (10 TB/s); software sees one GPU |
| Middle level | XCD: 32 CUs + L2 | GPC: 8 per GPU, 20 SMs each |
| Compute block | CU × 256 | SM × up to 160 |
| Matrix engines | 4 per CU = 1,024 | 4 Tensor Cores per SM = 640 |
| Lanes | 64 per CU = 16,384 | 128 CUDA cores per SM = 20,480 |
| Lockstep group | wavefront of 64 | warp of 32 |
| Scratch per block | LDS 160 KB | shared memory, plus 256 KB Tensor Memory per SM |
Same shape, different names. NVIDIA notes the enabled SM count varies by product.
Counts from AMD's CDNA 4 white paper and example rocminfo output; totals computed. Box layouts are schematic, not the real floorplan.
One package, many dies
Why chiplets: a single die cannot grow past the area a lithography machine exposes at once (the reticle limit, about 26 × 33 mm, roughly 850 mm²), and large dies have more defects. So vendors build a GPU from several dies on one package. Step through the data paths:
- Compute reads memoryAn XCD reads weights: the request goes through its IOD’s Infinity Cache and, on a miss, to the HBM stacks. In the default NPS1 mode addresses are spread over all 8 stacks.
- Across the IODsHalf of all addresses live on the other IOD’s stacks, so many accesses cross the IOD-to-IOD link. NPS2 mode keeps each half local.
- Host loads the modelThe host writes model weights into HBM over PCIe Gen5 x16, about 64 GB/s each way: done once at startup, not per token.
- GPU to GPUFor tensor parallelism a GPU exchanges partial results with the 7 others over Infinity Fabric (XGMI), 72 GB/s each way per link.
The chiplet boundaries are also where a GPU can be split: AMD’s compute partitions follow XCD lines and memory partitions follow IOD lines. See partitioning.
The spec sheet, side by side
All numbers are vendor peak or spec values, approximate.
| AMD Instinct MI355X | NVIDIA Blackwell Ultra | |
|---|---|---|
| Dies | 8 XCD + 2 IOD | 2 |
| Transistors | 185 billion | 208 billion |
| Compute blocks | 256 CUs | up to 160 SMs |
| Matrix engines | 1,024 matrix cores (4 per CU) | 640 Tensor Cores (4 per SM) |
| Lockstep group | wavefront, 64 lanes | warp, 32 threads |
| Peak clock | 2,400 MHz | not stated in the sources used here |
| Memory | 288 GB HBM3E, 8 TB/s, 8,192-bit | 288 GB HBM3E, 8 TB/s, 8,192-bit |
| Last-level cache | 256 MB Infinity Cache | not compared here |
| Power | 1,400 W typical board power | up to 1,400 W |
| Host link | PCIe Gen5 x16 | PCIe Gen6 x16; in GB300 also NVLink-C2C to a Grace CPU (900 GB/s) |
| GPU-to-GPU | 7 Infinity Fabric links, 72 GB/s each way per link | NVLink 5: 18 links, 1.8 TB/s total both ways |
| Dense FP8 | 5 PFLOPS | 5 PFLOPS |
| Dense 4-bit | 10.1 PFLOPS (MXFP4) | 15 PFLOPS (NVFP4) |
The two are close on memory, which is what decode cares about. The 4-bit formats are explained in number formats.
Reading it on a node
You can read this layout on an MI355X node without any special tool (example output):
$ rocminfo # the GPU agent, trimmed
Marketing Name: AMD Instinct MI355X
Compute Unit: 256
SIMDs per CU: 4
Shader Engines: 32
Wavefront Size: 64(0x40)
Max Waves Per CU: 32(0x20)
L2: 4096(0x1000) KB
L3: 262144(0x40000) KB
Max Clock Freq. (MHz): 2400
What to read in it: Compute Unit 256 and SIMDs per CU 4 are the zoom above. L2 is per XCD (4 MB × 8 = 32 MB); L3 is the 256 MB Infinity Cache. The 32 “shader engines” are the 4 CU arrays in each of the 8 XCDs. The kernel driver exposes the same in /sys/class/kfd/kfd/topology/nodes/<n>/properties: simd_count 1024, num_xcc 8, lds_size_in_kb 160 and gfx_target_version 90500 (gfx950, the instruction set name compilers target; see PTX and AMDGCN ISA).
Memory: HBM stacks
HBM (High Bandwidth Memory) is DRAM dies stacked on top of each other and placed millimetres from the compute dies on a silicon base layer. Very wide connections (1,024 bits per stack) at a moderate clock give huge bandwidth: 8 stacks × 1,024 bits × 8 Gb/s per pin ÷ 8 = 8,192 GB/s. amd-smi static on an MI355X node reports exactly that (example output):
VRAM:
TYPE: HBM3E
SIZE: 294896 MB
BIT_WIDTH: 8192
MAX_BANDWIDTH: 8192 GB/s
For comparison, one CPU socket on the same server has 12 DDR5 channels and roughly 0.5–0.6 TB/s (estimate, not measured). HBM is fast but small and fixed: you cannot add more to a GPU. Model weights plus the KV cache must fit in it. More in memory and interconnects.
Compute versus memory: why the pantry wins
Every generated token reads all the model’s weights from HBM. If the math is quick but memory is slow, the GPU waits: that work is memory-bound. Work that waits on math is compute-bound. Prefill is mostly compute-bound, decode mostly memory-bound. Peak throughput is just the parts multiplied; change the format or the clock and compare it with what HBM can feed:
A worked example: a 70-billion-parameter model in FP8 is 70 GB; at 8 TB/s, reading it once takes at least 8.75 ms, so one sequence can never decode faster than about 114 tokens per second on one such GPU, however many PFLOPS it has. LLM inference basics explains why this leads to disaggregated inference.
Power and heat
A 1,400 W part is a heater. MI355X comes in direct-liquid-cooled designs in AMD’s white paper; air-cooled variants report the board name Instinct MI355 AC OAM. The SMU firmware manages power and clocks all the time (example output):
$ amd-smi static -g 0 # LIMIT section, trimmed
PPT0: SOCKET_POWER_LIMIT: 1400 W
SLOWDOWN_HOTSPOT_TEMPERATURE: 100 °C
SLOWDOWN_VRAM_TEMPERATURE: 115 °C
SHUTDOWN_HOTSPOT_TEMPERATURE: 112 °C
SHUTDOWN_VRAM_TEMPERATURE: 125 °C
$ amd-smi metric -g 0 # an idle moment, trimmed
SOCKET_POWER: 250 W
GFX_0: CLK: 200 MHz MAX_CLK: 2400 MHz DEEP_SLEEP: ENABLED
MEM_0: CLK: 2000 MHz
What this means:
- Throttling is normal. Above the slowdown temperature or the power limit the GPU lowers its clock. Work gets slower, not wrong. AMD’s node problem detector rules (see GPU metrics) flag
AMDGPUJunctionTemperatureHighat 100 °C andAMDGPUMemoryTemperatureHighat 95 °C. - Shutdown is a hard stop. At the shutdown temperature the GPU protects itself; processes die.
- Idle is not zero. About 250 W with almost no work. Eight GPUs idle at about 2 kW per server.
- Clocks swing. About 200 MHz idle, up to 2,400 MHz busy. A single clock reading means little; use averages.
Host and device
The host is the server: CPUs, RAM, disks, NICs. The device is the GPU and its HBM. A program runs on the host, copies data to the device, starts kernels (functions that run on the GPU), and reads results back. The link between them is PCIe Gen5 x16: about 64 GB/s each way, more than 100 times slower than HBM. Good programs load weights once and keep them on the device.
An example 8-GPU OAM server has 2 AMD EPYC sockets (64 cores each, 256 hardware threads), 2 TiB RAM in 2 NUMA nodes, 8 MI355X and 8 400 Gb/s RDMA NICs. Each GPU hangs off one NUMA node; amd-smi static shows GPU 0 on NUMA node 0 with CPU cores 0–63 and 128–191 as local. A process that feeds a GPU from a CPU on a far NUMA node pays extra latency on every copy. Memory and interconnects and node topology show the layout.
A process “owns” GPU memory while it runs. amd-smi process (AMD) or nvidia-smi (NVIDIA) lists each process and the VRAM it holds. An inference server Pod typically holds most of the GPU’s HBM. How a GPU goes from power-on to running that process is in boot and firmware.
How this shows up in operations
- Kubernetes sees what the driver reports. AMD node labeller labels on an MI355X node include
amd.com/gpu.cu-count=256,amd.com/gpu.simd-count=1024andamd.com/gpu.vram=288G, and each ResourceSlice device carriescomputeUnits: 256andmemory: 294896Mi. If a number is lower, a partition mode changed or hardware is degraded. - One server is eight GPUs that share fate. Power, cooling, PCIe switches and the XGMI hive are shared; one bad GPU can take the others down in a reset.
- Temperature and power are leading indicators. A GPU that runs hot or throttles while its neighbours do not is a hardware ticket waiting to happen.
Common misconceptions
- “A GPU core is like a CPU core.” A “stream processor” is one SIMD lane, not a core with its own program counter. The CU or SM is the closest thing to a CPU core.
- “More TFLOPS means faster inference.” Decode is limited by HBM bandwidth.
- “GPU memory is like RAM; add more.” HBM is soldered on the package. Capacity is fixed per GPU model.
- “Two dies means two GPUs.” Blackwell’s two dies, and MI355X’s eight XCDs in SPX mode, appear as one device to software unless you partition.
Words to remember
- GPU, accelerator, device: used interchangeably here.
- VRAM: the GPU’s own memory (HBM on datacenter GPUs).
- Kernel: a function launched on the GPU. Not the Linux kernel; context tells which.
- OAM: the OCP Accelerator Module, the mezzanine form factor MI355X uses; eight sit on one baseboard (UBB).
- Driver: the Linux kernel module that talks to the GPU (
amdgpu,nvidia). See the software stack.
Further reading
- AMD CDNA 4 architecture white paper: XCDs, IODs, HBM, partitioning
- AMD Instinct MI355X specifications · ROCm GPU hardware specifications
- Inside NVIDIA Blackwell Ultra · NVIDIA GB300 NVL72
- rocminfo · amd-smi CLI
- The Illustrated Transformer: why LLM math is matrix multiplies