Learn · Overview
GPU fundamentals (overview)
This track is for a Linux and Kubernetes engineer who has never had to care what is inside a GPU. Read the pages in the order below; each builds on the one before. Every page puts AMD (MI355X) next to NVIDIA (Blackwell Ultra in HGX/DGX B300 and GB300 NVL72), and uses illustrative read-only output from an example 8-GPU MI355X server.
The idea first
Think of one GPU as a factory with its own warehouse, sitting in a business park (the server). The track answers, in order, the questions you would ask about such a factory:
| Question | Everyday version | Page |
|---|---|---|
| What is inside? | thousands of identical workstations next to a very fast warehouse | What a GPU is |
| How is the material packed? | smaller boxes ship faster but hold less detail | Number formats |
| How does it start up? | the factory’s own staff (firmware) open it before the head office (Linux driver) arrives | Boot and firmware |
| How does material arrive? | conveyor belts inside, roads between buildings, the motorway to other parks; each ten times slower | Memory and interconnects |
| Which road is nearest? | every factory has its own loading dock and its own on-ramp; use that one | Node topology |
| Who gives the orders? | the head office (host) owns the building rules; each tenant (container) brings its own tools | Software stack |
| Can tenants share it? | split the factory into sealed halves or eighths | Partitioning |
Where the picture breaks: a real factory runs many different jobs; a GPU runs the same instruction on many numbers at once, and is happiest when the job is one giant matrix multiply.
One prefill GPU, through the whole track
GPU 0 (0000:11:00.0) of an 8-GPU MI355X server, serving gpt-oss-120b
1 · The chip
- CUs
- 256
- 8 XCDs × 32
- HBM
- 288 GB
- 8 TB/s
- Weights
- ~65 GB
- 23% of HBM
- Held by the Pod
- ~248 GB
- the rest is KV cache reserve
The model fits on one MI355X with room to spare; vLLM reserves most of the rest for KV cache. What a GPU is.
2 · The numbers
Each token of the prompt leaves keys and values in every layer: 2 × 36 layers × 8 KV heads × 64 × 2 bytes = 72 KiB per token in BF16. A 4,096-token prompt: 302 MB. In FP8 it would be half. Number formats.
3 · The boot
At power-on the PSP boots from the card’s IFWI; at driver load amdgpu probes 0000:11:00.0, loads microcode, joins the 8-GPU XGMI hive and creates /dev/dri/renderD128 (plus the node-wide /dev/kfd). Boot and firmware.
4 · The links
| Move | Link | Time at peak |
|---|---|---|
| Load 65 GB of weights at startup | PCIe Gen5 x16, 64 GB/s | 1.02 s (once) |
| Read all weights once | HBM, 8 TB/s | 8.1 ms |
| Read one prompt’s KV (302 MB) | HBM | 38 µs |
| Ship that KV to another server | one 400 Gb/s NIC, 50 GB/s | 6.0 ms |
The network hop is 160× slower than HBM: disaggregated serving pays it once per request. Memory and interconnects.
5 · The layout
GPU 0 sits on NUMA node 0 (CPU cores 0–63, 128–191) under root pci0000:00, and shares a PCIe switch with NIC rdma0 = ens1np0. Do not assume NIC N belongs to GPU N: each name follows its own numbering. The KV bytes go GPU → switch → NIC without touching host RAM. Node topology.
6 · The software
Host: amdgpu driver 6.14.0 and firmware bundle 1.2.3 (example versions), shared by all 8 Pods. Image: ROCm build of vLLM, PyTorch, hipBLASLt, RCCL. A kernel launch is a packet written into a ring buffer plus a doorbell write; the driver does not see it. Software stack.
7 · The split
The card runs SPX + NPS1: one logical GPU with all 256 CUs and 288 GB. Kubernetes sees it as ResourceSlice device gpu-1-128 with partitionProfile: spx_nps1, one of 8 on the node. In CPX it would be 8 devices of 32 CUs each, and gpt-oss-120b’s ~65 GB of weights would no longer fit in one of them. Partitioning.
Example identifiers from the illustrative node `gpu-node-1`; times computed from approximate peak link speeds (real transfers are slower). Each step links to the page that explains it.
The picture in seven steps
- The chip. A GPU package is several dies: on MI355X, 8 compute dies (XCDs, 256 CUs) stacked on 2 I/O dies, with 8 HBM3E stacks (288 GB, 8 TB/s) beside them; on Blackwell Ultra, 2 large dies with up to 160 SMs and the same 288 GB and 8 TB/s. It draws up to 1,400 W, throttles when hot, and sits in a server as a PCIe “device” next to the “host” CPUs. → What a GPU is
- The numbers. Matrix engines run faster on narrower numbers: FP8 doubles BF16 throughput, and 4-bit formats with per-block scales (OCP MXFP4, NVIDIA NVFP4) double it again while halving memory. The format is a model-quality choice as much as a hardware one. → Number formats
- The boot. Firmware on the card (AMD IFWI with PSP and SMU; NVIDIA VBIOS with GSP) starts before Linux; the kernel driver then loads microcode, joins the GPU-to-GPU fabric, and creates device files. A kernel launch is a packet in a memory queue plus a doorbell write: the driver never sees it. → Boot and firmware
- The links. Bytes cross HBM (8 TB/s), then scale-up links between GPUs (XGMI full mesh of 8 on AMD, about 0.5 TB/s each way per GPU; NVLink through NVSwitch, 0.9 TB/s each way, up to 72 GPUs in NVL72), then PCIe (64 GB/s), then RDMA NICs (50–100 GB/s). Each step is roughly ten times slower and is its own fault domain: an AMD hive reset takes all 8 GPUs. → Memory and interconnects
- The layout. Each GPU on an 8-GPU MI355X server shares a PCIe switch, a NUMA node and one 400G NIC. Index numbers do not line up (GPU 0 pairs with
ens1np0, not a NIC numbered 0), so read the map withlspci, sysfs andamd-smi topology, ornvidia-smi topo -m. → Node topology - The software. Firmware, kernel driver and device files are on the host and shared; runtime, libraries and frameworks ship in the image. Versions must match across the boundary (ROCm compatibility matrix; CUDA minor and forward compatibility). Anything below the line means draining the node. → Software stack
- The split. One card can become several logical GPUs: AMD SPX to CPX along XCD lines and NPS1/NPS2 along IOD lines; NVIDIA MIG in sevenths and eighths. Large models use whole GPUs (SPX/NPS1); repartitioning is node maintenance and changes what “one GPU” means everywhere. → Partitioning
One MI355X server in numbers
An illustrative example server; versions are example versions.
| Item | Value |
|---|---|
| Server | 8-GPU OAM server, Ubuntu 24.04, kernel 6.8.0-example |
| CPUs | 2 × AMD EPYC (64 cores each), 2 NUMA nodes, 2 TiB RAM |
| GPUs | 8 × AMD Instinct MI355X (gfx950), 256 CUs and 288 GB HBM3E each, SPX/NPS1 |
| GPU links | one XGMI hive, full mesh, 72 GB/s each way per link |
| NICs | 8 × 400 Gb/s RoCE NICs, one per GPU on the same PCIe switch |
| Driver and firmware | amdgpu 6.14.0 (DKMS), PLDM firmware bundle 1.2.3 |
| Host ROCm | 7.0.0 (host tools only; Pods bring their own) |
| Kubernetes view | 8 devices in a gpu.amd.com ResourceSlice, partitionProfile: spx_nps1 |
AMD ↔ NVIDIA name map
| Idea | AMD | NVIDIA |
|---|---|---|
| Compute die | XCD (8 per MI355X) | die (2 per Blackwell) |
| I/O and cache die | IOD (2 per MI355X) | (on the same die) |
| Compute block | Compute Unit (CU) | Streaming Multiprocessor (SM) |
| Lockstep thread group | wavefront (64) | warp (32) |
| Matrix engine | Matrix core | Tensor Core |
| On-card security / management processor | PSP, SMU | GSP |
| Board firmware | IFWI in a PLDM bundle | VBIOS in a firmware bundle |
| GPU-to-GPU link | XGMI / Infinity Fabric | NVLink (+ NVSwitch) |
| Group of linked GPUs | hive | NVLink domain |
| Fabric control software | (none; driver) | Fabric Manager, NVLSM, IMEX |
| Kernel driver | amdgpu (+ KFD) |
nvidia (+ nvidia-uvm, nvidia-peermem) |
| Compute device file | /dev/kfd + /dev/dri/renderD* |
/dev/nvidiactl + /dev/nvidiaN |
| User-space runtime | ROCr + HIP | libcuda (host) + CUDA runtime |
| Programming model | HIP | CUDA |
| Software platform | ROCm | CUDA toolkit |
| Version rules | ROCm compatibility matrix | CUDA minor / forward compatibility |
| Collective library | RCCL | NCCL |
| Node CLI | amd-smi (older: rocm-smi) |
nvidia-smi |
| Topology view | amd-smi topology |
nvidia-smi topo -m |
| Health and metrics daemon | device-metrics-exporter (amd-smi library) | DCGM + dcgm-exporter |
| Hardware validation | RVS, AGFHC | DCGM diagnostics (dcgmi diag), field diag |
| Partitioning | compute (SPX…CPX) + memory (NPS) modes | MIG (+ MPS, time-slicing) |
| Retired memory | bad pages (amd-smi bad-pages) |
row remapping |
| Fault report | RAS / ECC events in the kernel log | Xid errors in the kernel log |
| Kubernetes operator | AMD GPU Operator (DeviceConfig) |
NVIDIA GPU Operator (ClusterPolicy) |
| DRA driver name | gpu.amd.com |
gpu.nvidia.com |
Where to go next
- How Kubernetes hands these GPUs to Pods: GPUs in Kubernetes.
- How GPUs fail and how to watch them: How GPUs fail, GPU metrics.
- How programs use the hardware: threads and wavefronts, memory and roofline.
- What runs on them: LLM inference basics.