Learn · Overview

GPU fundamentals (overview)

Updated 10 Nov 2026 · 7 min read

What this overview is built from
Kernels & siliconGPU fundamentalsBottom row: the smallest building blocks. Each line goes down from a page to a page it is built from; overviews (bold border) sit on top. Hover a box to light up what it is built from and what it is part of; click to open it.

This track is for a Linux and Kubernetes engineer who has never had to care what is inside a GPU. Read the pages in the order below; each builds on the one before. Every page puts AMD (MI355X) next to NVIDIA (Blackwell Ultra in HGX/DGX B300 and GB300 NVL72), and uses illustrative read-only output from an example 8-GPU MI355X server.

The idea first

Think of one GPU as a factory with its own warehouse, sitting in a business park (the server). The track answers, in order, the questions you would ask about such a factory:

Question Everyday version Page
What is inside? thousands of identical workstations next to a very fast warehouse What a GPU is
How is the material packed? smaller boxes ship faster but hold less detail Number formats
How does it start up? the factory’s own staff (firmware) open it before the head office (Linux driver) arrives Boot and firmware
How does material arrive? conveyor belts inside, roads between buildings, the motorway to other parks; each ten times slower Memory and interconnects
Which road is nearest? every factory has its own loading dock and its own on-ramp; use that one Node topology
Who gives the orders? the head office (host) owns the building rules; each tenant (container) brings its own tools Software stack
Can tenants share it? split the factory into sealed halves or eighths Partitioning

Where the picture breaks: a real factory runs many different jobs; a GPU runs the same instruction on many numbers at once, and is happiest when the job is one giant matrix multiply.

One prefill GPU, through the whole track

GPU 0 (0000:11:00.0) of an 8-GPU MI355X server, serving gpt-oss-120b

1 · The chip

CUs
256
8 XCDs × 32
HBM
288 GB
8 TB/s
Weights
~65 GB
23% of HBM
Held by the Pod
~248 GB
the rest is KV cache reserve

The model fits on one MI355X with room to spare; vLLM reserves most of the rest for KV cache. What a GPU is.

2 · The numbers

Each token of the prompt leaves keys and values in every layer: 2 × 36 layers × 8 KV heads × 64 × 2 bytes = 72 KiB per token in BF16. A 4,096-token prompt: 302 MB. In FP8 it would be half. Number formats.

3 · The boot

At power-on the PSP boots from the card’s IFWI; at driver load amdgpu probes 0000:11:00.0, loads microcode, joins the 8-GPU XGMI hive and creates /dev/dri/renderD128 (plus the node-wide /dev/kfd). Boot and firmware.

4 · The links

MoveLinkTime at peak
Load 65 GB of weights at startupPCIe Gen5 x16, 64 GB/s1.02 s (once)
Read all weights onceHBM, 8 TB/s8.1 ms
Read one prompt’s KV (302 MB)HBM38 µs
Ship that KV to another serverone 400 Gb/s NIC, 50 GB/s6.0 ms

The network hop is 160× slower than HBM: disaggregated serving pays it once per request. Memory and interconnects.

5 · The layout

GPU 0 sits on NUMA node 0 (CPU cores 0–63, 128–191) under root pci0000:00, and shares a PCIe switch with NIC rdma0 = ens1np0. Do not assume NIC N belongs to GPU N: each name follows its own numbering. The KV bytes go GPU → switch → NIC without touching host RAM. Node topology.

6 · The software

Host: amdgpu driver 6.14.0 and firmware bundle 1.2.3 (example versions), shared by all 8 Pods. Image: ROCm build of vLLM, PyTorch, hipBLASLt, RCCL. A kernel launch is a packet written into a ring buffer plus a doorbell write; the driver does not see it. Software stack.

7 · The split

The card runs SPX + NPS1: one logical GPU with all 256 CUs and 288 GB. Kubernetes sees it as ResourceSlice device gpu-1-128 with partitionProfile: spx_nps1, one of 8 on the node. In CPX it would be 8 devices of 32 CUs each, and gpt-oss-120b’s ~65 GB of weights would no longer fit in one of them. Partitioning.

Example identifiers from the illustrative node `gpu-node-1`; times computed from approximate peak link speeds (real transfers are slower). Each step links to the page that explains it.

The picture in seven steps

  1. The chip. A GPU package is several dies: on MI355X, 8 compute dies (XCDs, 256 CUs) stacked on 2 I/O dies, with 8 HBM3E stacks (288 GB, 8 TB/s) beside them; on Blackwell Ultra, 2 large dies with up to 160 SMs and the same 288 GB and 8 TB/s. It draws up to 1,400 W, throttles when hot, and sits in a server as a PCIe “device” next to the “host” CPUs. → What a GPU is
  2. The numbers. Matrix engines run faster on narrower numbers: FP8 doubles BF16 throughput, and 4-bit formats with per-block scales (OCP MXFP4, NVIDIA NVFP4) double it again while halving memory. The format is a model-quality choice as much as a hardware one. → Number formats
  3. The boot. Firmware on the card (AMD IFWI with PSP and SMU; NVIDIA VBIOS with GSP) starts before Linux; the kernel driver then loads microcode, joins the GPU-to-GPU fabric, and creates device files. A kernel launch is a packet in a memory queue plus a doorbell write: the driver never sees it. → Boot and firmware
  4. The links. Bytes cross HBM (8 TB/s), then scale-up links between GPUs (XGMI full mesh of 8 on AMD, about 0.5 TB/s each way per GPU; NVLink through NVSwitch, 0.9 TB/s each way, up to 72 GPUs in NVL72), then PCIe (64 GB/s), then RDMA NICs (50–100 GB/s). Each step is roughly ten times slower and is its own fault domain: an AMD hive reset takes all 8 GPUs. → Memory and interconnects
  5. The layout. Each GPU on an 8-GPU MI355X server shares a PCIe switch, a NUMA node and one 400G NIC. Index numbers do not line up (GPU 0 pairs with ens1np0, not a NIC numbered 0), so read the map with lspci, sysfs and amd-smi topology, or nvidia-smi topo -m. → Node topology
  6. The software. Firmware, kernel driver and device files are on the host and shared; runtime, libraries and frameworks ship in the image. Versions must match across the boundary (ROCm compatibility matrix; CUDA minor and forward compatibility). Anything below the line means draining the node. → Software stack
  7. The split. One card can become several logical GPUs: AMD SPX to CPX along XCD lines and NPS1/NPS2 along IOD lines; NVIDIA MIG in sevenths and eighths. Large models use whole GPUs (SPX/NPS1); repartitioning is node maintenance and changes what “one GPU” means everywhere. → Partitioning

One MI355X server in numbers

An illustrative example server; versions are example versions.

Item Value
Server 8-GPU OAM server, Ubuntu 24.04, kernel 6.8.0-example
CPUs 2 × AMD EPYC (64 cores each), 2 NUMA nodes, 2 TiB RAM
GPUs 8 × AMD Instinct MI355X (gfx950), 256 CUs and 288 GB HBM3E each, SPX/NPS1
GPU links one XGMI hive, full mesh, 72 GB/s each way per link
NICs 8 × 400 Gb/s RoCE NICs, one per GPU on the same PCIe switch
Driver and firmware amdgpu 6.14.0 (DKMS), PLDM firmware bundle 1.2.3
Host ROCm 7.0.0 (host tools only; Pods bring their own)
Kubernetes view 8 devices in a gpu.amd.com ResourceSlice, partitionProfile: spx_nps1

AMD ↔ NVIDIA name map

Idea AMD NVIDIA
Compute die XCD (8 per MI355X) die (2 per Blackwell)
I/O and cache die IOD (2 per MI355X) (on the same die)
Compute block Compute Unit (CU) Streaming Multiprocessor (SM)
Lockstep thread group wavefront (64) warp (32)
Matrix engine Matrix core Tensor Core
On-card security / management processor PSP, SMU GSP
Board firmware IFWI in a PLDM bundle VBIOS in a firmware bundle
GPU-to-GPU link XGMI / Infinity Fabric NVLink (+ NVSwitch)
Group of linked GPUs hive NVLink domain
Fabric control software (none; driver) Fabric Manager, NVLSM, IMEX
Kernel driver amdgpu (+ KFD) nvidia (+ nvidia-uvm, nvidia-peermem)
Compute device file /dev/kfd + /dev/dri/renderD* /dev/nvidiactl + /dev/nvidiaN
User-space runtime ROCr + HIP libcuda (host) + CUDA runtime
Programming model HIP CUDA
Software platform ROCm CUDA toolkit
Version rules ROCm compatibility matrix CUDA minor / forward compatibility
Collective library RCCL NCCL
Node CLI amd-smi (older: rocm-smi) nvidia-smi
Topology view amd-smi topology nvidia-smi topo -m
Health and metrics daemon device-metrics-exporter (amd-smi library) DCGM + dcgm-exporter
Hardware validation RVS, AGFHC DCGM diagnostics (dcgmi diag), field diag
Partitioning compute (SPX…CPX) + memory (NPS) modes MIG (+ MPS, time-slicing)
Retired memory bad pages (amd-smi bad-pages) row remapping
Fault report RAS / ECC events in the kernel log Xid errors in the kernel log
Kubernetes operator AMD GPU Operator (DeviceConfig) NVIDIA GPU Operator (ClusterPolicy)
DRA driver name gpu.amd.com gpu.nvidia.com

Where to go next

Further reading