Learn · GPU fundamentals
The GPU software stack
A call like torch.matmul on a GPU passes through six layers of software before the hardware does anything. For operations, one split decides almost everything: the lower three layers live on the host and are shared by every Pod; the upper three ship inside each container image. The first group changes only in a maintenance window that stops all GPU work on the node. The second changes with every image rollout.
The idea first
Picture a restaurant. The dining room (the container image) is redecorated whenever the owner likes: menu, waiters, recipes. The kitchen equipment and gas line (the host’s driver and firmware) is shared by every table and only changes when the restaurant closes for maintenance.
| Layer | Restaurant picture | The question it answers |
|---|---|---|
| Framework (PyTorch, vLLM) | the customer’s order | “What maths does the model need?” |
| Libraries (hipBLASLt, RCCL) | the recipe book | “Which GPU program does this maths fastest, for this shape?” |
| Runtime (HIP, ROCr) | the waiter who writes the ticket | “How do I hand the GPU a program and its inputs?” |
| Device files | the kitchen door | “Who may talk to the GPU at all?” |
| Kernel driver | the kitchen manager | “Who owns which memory and which queue?” |
| Firmware | the stove’s own controls | “Is the hardware powered, trusted, and working?” |
Where the picture breaks: after the first order, the waiter does not walk through the manager any more. The runtime pins a ticket rail (a queue in memory that the GPU reads) and a bell (a doorbell) once, then drops every order straight onto the rail. The driver is needed to set things up, not for each launch.
Pick a vendor to see which layer comes from where. The one real difference is NVIDIA’s libcuda.so, the user-space half of the driver: it must come from the host.
Worked example: one matrix multiply, from PyTorch to the doorbell
Follow one real-sized call: the query projection of gpt-oss-120b over a 4,085-token prompt (hidden size 2,880 in, 64 heads × 64 = 4,096 out), in BF16, on one MI355X in SPX mode. The tile size (256 × 256) and 256 threads per workgroup are illustrative: the library picks its own kernel. Every number below is computed from the shape.
q = x @ W_q on one MI355X
1 · PyTorch: a matrix multiply on a GPU tensor
- x (activations)
- 23.5 MB
- 4085 × 2880 BF16
- W_q (weights)
- 23.6 MB
- 2880 × 4096 BF16
- q (output)
- 33.5 MB
- 4085 × 4096 BF16
- Work
- 96.4 GFLOP
- 2 × M × N × K
PyTorch sees two tensors that live in GPU memory (the ROCm build reuses the torch.cuda names, so the device is cuda:0) and dispatches the multiply to its GPU implementation, which calls the BLAS library. Python never touches the numbers; it only decides what to run.
2 · hipBLASLt: split the output into 256 tiles
| rows 0–255 | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rows 256–511 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 |
| rows 512–767 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 |
| rows 768–1023 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 |
| rows 1024–1279 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 |
| rows 1280–1535 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95 |
| rows 1536–1791 | 96 | 97 | 98 | 99 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | 107 | 108 | 109 | 110 | 111 |
| rows 1792–2047 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | 124 | 125 | 126 | 127 |
| rows 2048–2303 | 128 | 129 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143 |
| rows 2304–2559 | 144 | 145 | 146 | 147 | 148 | 149 | 150 | 151 | 152 | 153 | 154 | 155 | 156 | 157 | 158 | 159 |
| rows 2560–2815 | 160 | 161 | 162 | 163 | 164 | 165 | 166 | 167 | 168 | 169 | 170 | 171 | 172 | 173 | 174 | 175 |
| rows 2816–3071 | 176 | 177 | 178 | 179 | 180 | 181 | 182 | 183 | 184 | 185 | 186 | 187 | 188 | 189 | 190 | 191 |
| rows 3072–3327 | 192 | 193 | 194 | 195 | 196 | 197 | 198 | 199 | 200 | 201 | 202 | 203 | 204 | 205 | 206 | 207 |
| rows 3328–3583 | 208 | 209 | 210 | 211 | 212 | 213 | 214 | 215 | 216 | 217 | 218 | 219 | 220 | 221 | 222 | 223 |
| rows 3584–3839 | 224 | 225 | 226 | 227 | 228 | 229 | 230 | 231 | 232 | 233 | 234 | 235 | 236 | 237 | 238 | 239 |
| rows 3840–4084 | 240 | 241 | 242 | 243 | 244 | 245 | 246 | 247 | 248 | 249 | 250 | 251 | 252 | 253 | 254 | 255 |
The last tile row covers only 245 of 256 rows: 99.7% of the tile area is useful work.
The library picks a kernel (a compiled GPU function in the image, built for gfx950) that suits this shape. Each workgroup (a team of threads that share fast LDS memory) computes one 256 × 256 tile of the output, reading a 256-row strip of x and a 256-column strip of Wq. 16 × 16 = 256 workgroups: on MI355X, exactly one per compute unit.
3 · HIP: pack the arguments, point at the code
The HIP runtime copies the kernel’s arguments (the GPU addresses of x, Wq and q, and M, N, K) into a small kernarg buffer in memory the GPU can read, and looks up the kernel object: the address of the compiled code’s descriptor. Two pointers in the packet carry them.
4 · ROCr: write a 64-byte AQL packet into the ring
hsa_kernel_dispatch_packet_t (ROCR-Runtime hsa.h). Values: this page's example launch.The runtime fills the next free slot of the process’s user-mode queue, a ring buffer of 64-byte packets in memory. grid_size_x counts threads, not workgroups: 256 × 256 = 65,536. group_segment_size is the LDS each workgroup needs (64 KiB here, of the CU’s 160 KB; also illustrative). The header is written last, so the GPU never reads a half-written packet.
5 · Ring the doorbell: no system call
- System calls
- 0
- Packet
- 64 bytes
- one ring slot
- Doorbell
- 1 store
- the new write index
The runtime writes the queue’s new write index to the doorbell, a memory-mapped register on the GPU that the driver mapped into the process when the queue was created. This is an ordinary store instruction. The Linux kernel and the amdgpu driver do not see this launch at all, which is why a launch costs microseconds and why a hung kernel shows up only as a queue that stops moving.
6 · Command processor: 256 workgroups over 8 XCDs
| 1st | 2nd | 3rd | 4th | 5th | |
|---|---|---|---|---|---|
| XCD 0 | 0 | 8 | 16 | 24 | 32 |
| XCD 1 | 1 | 9 | 17 | 25 | 33 |
| XCD 2 | 2 | 10 | 18 | 26 | 34 |
| XCD 3 | 3 | 11 | 19 | 27 | 35 |
| XCD 4 | 4 | 12 | 20 | 28 | 36 |
| XCD 5 | 5 | 13 | 21 | 29 | 37 |
| XCD 6 | 6 | 14 | 22 | 30 | 38 |
| XCD 7 | 7 | 15 | 23 | 31 | 39 |
…and so on: each XCD gets 32 workgroups, one per CU (32 CUs per XCD) in an even spread.
The GPU’s command processor reads the packet and hands workgroups out round-robin over the 8 XCDs (AMD documents this for SPX mode): workgroup 0 to XCD 0, 1 to XCD 1, … 8 back to XCD 0. Inside each XCD, a workgroup’s 256 threads run as 4 wavefronts of 64 on one CU, on its matrix cores. Neighbouring tiles land on different XCDs, so they do not share an L2 cache: one reason kernels for MI355X care about workgroup order.
7 · Done: the completion signal
- Floor at BF16 peak
- 38.6 µs
- 96.4 GFLOP at ~2.5 PFLOPS
- Workgroups
- 256
- 1 per CU
- Wavefronts
- 1024
- 4 per workgroup
When the last workgroup finishes, the command processor decrements the packet’s completion signal, a value in memory. The host thread (or the next packet, through a barrier bit) waits on it. Even at the vendor’s peak rate this multiply takes at least 38.6 µs; real kernels reach a fraction of peak. One forward pass of a model is many such launches (several per layer), which is why serving engines such as vLLM can capture them into graphs (CUDA graphs, HIP graphs on ROCm) and replay them with less launch overhead.
M = 4085, K = 2880, N = 4096, BF16; tile 256 × 256 and 256 threads per workgroup are assumptions for illustration.
Change the shape and see how the launch changes. Short prompts or narrow outputs leave most of the 256 CUs idle: one reason a single prefill uses the GPU badly and batching helps.
The layers on an 8-GPU MI355X server
An illustrative example node (Ubuntu 24.04, kernel 6.8.0-example); every version below is a made-up example:
| Layer | AMD, as installed | NVIDIA equivalent | Where | Changed by |
|---|---|---|---|---|
| Firmware on the card | IFWI 113-EXAMPLE-0001, PLDM bundle 01.00.00.00 |
VBIOS, HGX/NVL72 firmware bundle | GPU flash | flashing via BMC, then a cold power cycle |
| Firmware loaded by the driver | amdgpu-dkms-firmware 30.10.0 (several hundred microcode files in /lib/firmware/updates/amdgpu/) |
GSP firmware gsp_*.bin, part of the driver release |
host disk | package update, then driver reload |
| Kernel driver | amdgpu 6.14.0.30100000 = AMD GPU Driver 30.10.0, built by DKMS |
nvidia, nvidia-uvm, nvidia-modeset, nvidia-drm, nvidia-peermem (open modules) |
host kernel | package + rebuild, module reload or reboot |
| Device files | /dev/kfd, /dev/dri/renderD* |
/dev/nvidiactl, /dev/nvidia0…, /dev/nvidia-uvm, /dev/nvidia-caps/* |
host, passed into containers | driver |
| Runtime | ROCr (HSA runtime) + HIP runtime | libcuda.so (driver API) + libcudart (CUDA runtime) |
container (AMD); NVIDIA libcuda comes from the host |
image |
| Libraries | hipBLASLt, rocBLAS, RCCL, MIOpen, Composable Kernel | cuBLAS(Lt), cuDNN, NCCL, CUTLASS | container | image |
| Frameworks | PyTorch (ROCm build), vLLM | PyTorch (CUDA build), vLLM, TensorRT-LLM | container | image |
The host also has ROCm 7.0.0 installed as packages (for example amd-smi, RCCL and RVS). Those serve host tools like amd-smi and rocminfo; Pods do not use them. amd-smi version prints the whole chain in one line:
AMDSMI Tool: 26.0.0+example | AMDSMI Library version: 26.0.0 | ROCm version: 7.0.0 | amdgpu version: 6.14.0.30100000
Firmware
Covered in depth in boot and firmware. The key point for the stack: AMD “firmware” is two artefacts. The IFWI/PLDM bundle lives on the card and the PSP boots from it. The microcode package lives on the host and the driver hands it to the PSP at every driver load. Both appear in the ROCm compatibility matrix. On NVIDIA, the GSP firmware ships with each driver release and must match the kernel modules exactly.
Kernel driver
AMD. One module, amdgpu, drives the hardware. Inside it, KFD (the kernel fusion driver) is the compute interface ROCm uses: /dev/kfd, user-mode queues, the topology in /sys/class/kfd. On a typical install amdgpu comes from AMD’s amdgpu-dkms package, not from the distribution kernel, so the GPU driver can be newer than the distribution kernel. lsmod shows the package brings its own copies of shared graphics helpers (amdttm, amddrm_buddy, amd_sched, amdxcp, and amdkcl, a compatibility layer for older kernels).
DKMS (Dynamic Kernel Module Support) compiles out-of-tree modules for each installed kernel. dkms status on the example node:
amdgpu/6.14.0-example, 6.8.0-example-generic, x86_64: installed (Original modules exist) ...
mlnx-ofa_kernel/24.10.example, 6.8.0-example-generic, x86_64: installed ...
The NVIDIA ConnectX stack (mlnx-ofa_kernel, xpmem, srp, iser) is built the same way. The consequence: a kernel update must rebuild every DKMS module before the next boot, or the node comes up without a GPU or RDMA driver. (dkms status also prints “Differences between built and installed modules” for these modules.)
Driver behaviour is tuned by module parameters and the kernel command line. A typical MI355X host carries amdgpu.noretry=1 (GPU page faults are not retried), iommu=pt (IOMMU in pass-through for host devices), numa_balancing=disable and pcie_aspm=off (no PCIe link power saving). Current values are in /sys/module/amdgpu/parameters/.
NVIDIA. The driver is several modules: nvidia.ko (core), nvidia-uvm.ko (unified memory), nvidia-modeset.ko and nvidia-drm.ko (display, also used for some memory sharing), and nvidia-peermem.ko (lets RDMA NICs access GPU memory). NVIDIA now ships open kernel modules (MIT/GPLv2). Blackwell requires them: “the proprietary drivers are unsupported on these platforms”. On Ubuntu and RHEL the package is nvidia-open, installed through DKMS or as precompiled modules.
Device files and the container boundary
User space reaches the driver only through character devices (listed in boot and firmware). A container sees a GPU only if its device files are added to the container. On current Kubernetes that typically happens with CDI (Container Device Interface): when a Pod starts, the DRA driver writes a small spec under /var/run/cdi/ naming the device files (and, on NVIDIA, host libraries) to inject; containerd reads it when it creates the container. After that the container uses the GPU directly; the DRA driver is not on the data path. See DRA on the node.
On the host, /dev/kfd is crw-rw-rw- and group render; the render nodes are group render too. Inside a container, the process must be allowed to open them (device cgroup and group membership), which CDI takes care of.
The big AMD/NVIDIA difference at this boundary (the vendor explorer above): on AMD everything above the device files is in the image, and the host contributes only the kernel driver and device files. On NVIDIA, libcuda.so must match the kernel module exactly, so the NVIDIA Container Toolkit (or the CDI spec it generates) mounts the host’s libcuda.so, nvidia-smi and related libraries into each container; the image brings the CUDA runtime (libcudart) and libraries.
Runtime, libraries, frameworks
- ROCr is AMD’s implementation of the HSA runtime. It opens
/dev/kfd, enumerates agents (the CPUs and GPUs thatrocminfoprints), allocates memory and creates queues. - HIP is the CUDA-like programming interface on top.
hipMalloc,hipMemcpyand kernel launches go through it. Code is compiled for a specific GPU target (gfx950for MI355X); an image withoutgfx950code objects will not run on an MI355X. See PTX and AMDGCN ISA. - CUDA has two user-space APIs: the driver API in
libcuda(from the host) and the runtime API inlibcudart(from the image). NVIDIA code usually ships PTX as well as compiled machine code, so the driver can compile for newer GPUs at load time. - Libraries do the heavy lifting: BLAS for matrix multiplies (hipBLASLt, cuBLASLt), collectives for multi-GPU communication (RCCL, NCCL), attention kernels.
- Frameworks: PyTorch has separate ROCm and CUDA builds; vLLM runs on both.
Compatibility: which versions may meet
AMD: the ROCm compatibility matrix
Each ROCm release publishes a matrix of supported GPUs, OS kernels, AMD GPU Driver versions and PLDM firmware versions. An entry for the MI350 series has this shape (illustrative values for the example ROCm 7.0.0):
- a range of drivers, for example
30.10.0(newest) back to30.0.0; - a minimum PLDM bundle, for example
01.00.00.00or later; - the supported OS releases and kernels, for example Ubuntu 24.04.
The example node runs driver 30.10.0, PLDM 01.00.00.00 and Ubuntu 24.04: on the matrix. The driver version appears twice in different forms: 30.10.0 in AMD’s documents and 6.14.0.30100000 in modinfo, package names and the DRA attribute driverVersionFull. The suffix 30100000 encodes 30.10.0.
The rule for containers: the ROCm version inside the image must be supported with the driver on the host. Several ROCm versions can run side by side in different Pods as long as each one lists the host driver. Newer ROCm releases list newer drivers, so an image upgrade to a much newer ROCm may need a host driver upgrade first.
NVIDIA: minor version and forward compatibility
NVIDIA defines three ways a CUDA application meets a driver:
- Backward compatibility: an application built with an older CUDA toolkit runs on a newer driver. Always.
- Minor version compatibility (since CUDA 11): an application built with any CUDA 13.x runs on any driver of release 580 or later, “with limited feature-set”. NVIDIA’s example runs a CUDA 13.4 application on driver 580.65.06, which shipped with CUDA 13.0.
- Forward compatibility (data center GPUs only): the
cuda-compat-<major>-<minor>package contains a newerlibcuda.sothat works with an older kernel module, so an application built for a newer CUDA major release runs on an older driver. You put it onLD_LIBRARY_PATH. For example,cuda-compat-13-5contains the 620 driver libraries.
A system that pins a 580-series driver with CUDA 13.0, for example, still runs CUDA 13.x images through minor version compatibility. Two more NVIDIA rules: the open kernel modules must be used with GSP firmware and user-space libraries from the same driver release, and for drivers before 620 the Fabric Manager version must match the driver version.
Who installs what
- Host layers. Firmware, OS, kernel,
amdgpu-dkms, ROCm host tools and the NIC stack are usually installed by the provisioning system, not by Kubernetes. The AMD GPU Operator can manage the driver through KMM, but it can also run with driver management turned off (driver.enable: false), so the operator never loads or replaces the kernel module. - What Kubernetes adds. The operator’s DRA driver, node labeller and metrics exporter, all running as Pods, all user space.
- NVIDIA differs. NVIDIA’s GPU Operator can install the driver as a container on the node, or use a driver preinstalled on the host (for example DGX OS).
What an upgrade disrupts
Least to most disruptive:
| Component | Disrupts running GPU work? | Reach |
|---|---|---|
| DRA driver | no for running work | new claims on one node at a time |
| GPU operator | normally no | — |
| ROCm / user space on the host | host tools only; Pods bring their own | one node |
| GPU kernel driver | yes, module reload or reboot | every GPU on the node |
| GPU firmware | yes, reboot plus a full power cycle | every GPU on the node |
| Host kernel, NIC, OS | yes, reboot | every GPU on the node |
A kernel module cannot be unloaded while any process holds the GPU, and an AMD driver reload resets every GPU in the hive. So every row from the driver down means: drain the node’s GPU workloads first.
Common misconceptions
- “The container brings its own driver.” It brings user space only. The kernel module is shared and must be compatible with every image on the node.
- “
nvidia-smishows the CUDA version installed.” It shows the newest CUDA version the driver supports, not the toolkit in any image. - “ROCm version equals driver version.” They are separate products with a support matrix between them (ROCm 7.0.0 and driver 30.10.0 in the example).
- “Upgrading the image is always safe.” Not if the new ROCm or CUDA needs a newer host driver than the node has.
Further reading
- ROCm compatibility matrix
- AMD GPU driver (amdgpu) documentation · HIP · amdgpu driver core (kernel docs)
- CUDA compatibility · minor version compatibility · forward compatibility
- NVIDIA driver kernel modules · NVIDIA Container Toolkit
- Container Device Interface
- ROCR-Runtime
hsa.h:hsa_kernel_dispatch_packet_t(the AQL packet fields)