Learn · GPUs in Kubernetes
CDI: how a GPU gets into a container
Allocation decides which GPU a Pod gets. Something still has to put that GPU’s device files into the container, and only those. With a DRA driver and containerd 2.x, that something is CDI, the Container Device Interface.
The idea
A container sees only the device files the runtime puts into it. Think of a CDI spec as a recipe card pinned on a board (/var/run/cdi). The card has a name, k8s.gpu.amd.com/gpu=<claim>-gpu-5-132, and says “to cook this, add these three device files and these two environment variables”. The DRA driver writes the card. The kubelet only passes the name along. containerd looks the name up on the board and follows the recipe when it builds the container.
Each part answers one question:
| Piece | Question it answers |
|---|---|
| CDI device name | Which device does this container want? (a string, nothing more) |
Spec file in /var/run/cdi |
What does that name mean on this node? (device nodes, env, mounts) |
| containerd’s CDI cache | How do I turn names into an OCI spec? |
| Device cgroup rule | Enforce: the container may open these major:minor numbers and no others |
Where the analogy breaks: the card is read once, at container create. Nobody re-reads it while the container runs, so deleting the file later does not take the GPU away, and the driver that wrote it is not in the data path. But every container restart is a new create, so the card must still be on the board then.
Before CDI, every vendor shipped its own runtime hook or wrapper (for example nvidia-container-runtime). With CDI the runtime does the work and the vendor only writes data.
Worked example: claim c1a10001 becomes three device files
Follow the prefill Pod …-p4x7k on gpu-node-1, which the scheduler gave gpu-5-132. Every file below is illustrative example output in the driver’s format; the derived values (file name, minor numbers, OCI edits) are computed from the device name at build time and checked against the example file.
From NodePrepareResources to /dev/dri/renderD132
1 · The kubelet asks the driver to prepare the claim
# NodePrepareResources request (gRPC, dra.sock)
claims:
- namespace: gpu-demo
uid: c1a10001-0000-4000-8000-000000000001
name: llm-prefill-5f7d8c9b6-p4x7k-gpu-z8k3qThe kubelet sends only the claim’s identity. The driver reads the claim from the API and finds its own result in status.allocation: driver gpu.amd.com, pool gpu-node-1, device gpu-5-132.
2 · The driver writes the CDI spec file
# /var/run/cdi/k8s.gpu.amd.com-gpu_c1a10001-0000-4000-8000-000000000001.yaml
cdiVersion: 0.5.0
kind: k8s.gpu.amd.com/gpu
devices:
- name: c1a10001-0000-4000-8000-000000000001-gpu-5-132
containerEdits:
deviceNodes:
- path: /dev/kfd
hostPath: /dev/kfd
type: c
major: 238
permissions: rwm
- path: /dev/dri/card5
hostPath: /dev/dri/card5
type: c
major: 226
minor: 5
permissions: rwm
- path: /dev/dri/renderD132
hostPath: /dev/dri/renderD132
type: c
major: 226
minor: 132
permissions: rwmFile name = kind k8s.gpu.amd.com/gpu with / → -, then _<claim uid>.yaml: /var/run/cdi/k8s.gpu.amd.com-gpu_c1a10001-0000-4000-8000-000000000001.yaml. Device name gpu-5-132 is gpu-<card>-<render>, so the GPU’s nodes are card5 and renderD132, both DRM major 226. /dev/kfd has no minor in the file; the CDI library fills it by stat-ing the host file (238:0).
3 · …and keeps one shared file for every GPU container
# /var/run/cdi/k8s.gpu.amd.com-gpu_common.yaml
cdiVersion: 0.3.0
kind: k8s.gpu.amd.com/gpu
devices:
- name: common
containerEdits:
env:
- KUBERNETES_NODE_NAME=gpu-node-1
- DRA_RESOURCE_DRIVER_NAME=gpu.amd.comWritten once when the driver started (1 Nov 08:00). The driver answers the kubelet with two CDI names: k8s.gpu.amd.com/gpu=common and k8s.gpu.amd.com/gpu=c1a10001-0000-4000-8000-000000000001-gpu-5-132.
4 · The kubelet passes the names in CRI CreateContainer
# CRI CreateContainerRequest → config (only the GPU part)
config:
metadata: {name: prefill}
CDI_devices:
- name: k8s.gpu.amd.com/gpu=common
- name: k8s.gpu.amd.com/gpu=c1a10001-0000-4000-8000-000000000001-gpu-5-132The kubelet stores the names in its checkpoint and puts them in the CRI field ContainerConfig.CDI_devices every time it creates a container that lists the claim. It never sees device numbers.
5 · containerd resolves the names against its spec dirs
# /etc/containerd/config.toml (excerpt)
version = 3
enable_cdi = true
cdi_spec_dirs = ['/etc/cdi', '/var/run/cdi']| Name part | Value | Found in |
|---|---|---|
| kind (vendor/class) | k8s.gpu.amd.com/gpu |
both files |
device common |
2 env vars | k8s.gpu.amd.com-gpu_common.yaml |
device c1a10001…-gpu-5-132 |
3 device nodes | k8s.gpu.amd.com-gpu_c1a10001….yaml |
/etc/cdi does not exist on GPU nodes; everything is in /var/run/cdi. An unknown name fails the create.
6 · The edits land in the OCI runtime spec
"process": {"env": [ …, "KUBERNETES_NODE_NAME=gpu-node-1", "DRA_RESOURCE_DRIVER_NAME=gpu.amd.com"]},
"linux": {
"devices": [
{"path": "/dev/kfd", "type": "c", "major": 238, "minor": 0},
{"path": "/dev/dri/card5", "type": "c", "major": 226, "minor": 5},
{"path": "/dev/dri/renderD132", "type": "c", "major": 226, "minor": 132}
],
"resources": {"devices": [
{"allow": false, "access": "rwm"}, // containerd default: deny all
{"allow": true, "type": "c", "major": 238, "minor": 0, "access": "rwm"},
{"allow": true, "type": "c", "major": 226, "minor": 5, "access": "rwm"},
{"allow": true, "type": "c", "major": 226, "minor": 132, "access": "rwm"}
]}
}For each device node the CDI library adds a linux.devices entry (runc creates the file) and a device-cgroup allow rule (the kernel lets the container open it). Env entries are appended to the process environment.
7 · The container sees one GPU
/dev/kfd c 238,0 # ROCm compute interface (one for all GPUs)
/dev/dri/card5 c 226,5 # DRM card node of GPU 0000:8b:00.0
/dev/dri/renderD132 c 226,132 # render node of the same GPU
KUBERNETES_NODE_NAME=gpu-node-1
DRA_RESOURCE_DRIVER_NAME=gpu.amd.comrunc creates the container at 10:12:00 (node clock). The cgroup (a BPF device filter on cgroup v2) allows 238:0, 226:5 and 226:132 and nothing else, so the other 7 GPUs’ render nodes cannot be opened even if someone creates them. The model then loads for 4 min 0 s before the Pod turns Ready.
Claim UID c1a10001-0000-4000-8000-000000000001, containerd 2.2, AMD DRA driver v1.0.1 (example).
The same path, hop by hop
- PrepareThe kubelet calls
NodePrepareResources; the reply carries the two CDI names. - Write the specDuring prepare the driver writes
k8s.gpu.amd.com-gpu_c1a10001….yamlinto/var/run/cdi. - CreateThe kubelet sends CRI
CreateContainerwithCDI_devices= the two names. - Resolvecontainerd’s CDI cache looks the names up in
/etc/cdiand/var/run/cdi. - Applycontainerd adds the edits to the OCI spec; runc makes the device nodes and the cgroup rules.
- Failure: spec file missingIf the file is gone at create time (driver lost it, or
/var/runwas wiped), the name does not resolve, containerd refuses the container and the Pod showsCreateContainerError. - Driver gone, container fineOnce the container exists the driver is out of the picture. It keeps its device files even if the driver Pod dies.
The spec format
A spec file has cdiVersion (for example 0.5.0), kind (vendor/class), and devices[], each with a name and containerEdits. containerEdits (per device, or top-level for all devices of the kind) holds env, deviceNodes, mounts, hooks, and since 0.7.0 additionalGids and intelRdt. A device’s fully qualified name is <kind>=<name>.
Runtimes look in /etc/cdi (static, written by packages) and /var/run/cdi (dynamic, written at run time). Files in /var/run/cdi win when names collide. On gpu-node-1 there is one common file plus one file per prepared claim, named after the claim UID:
$ ls -l /var/run/cdi # example output
-rw-------. 1 root root 210 Nov 1 08:00 k8s.gpu.amd.com-gpu_common.yaml
-rw-------. 1 root root 660 Nov 3 10:12 k8s.gpu.amd.com-gpu_c1a10001-0000-4000-8000-000000000001.yaml
-rw-------. 1 root root 660 Nov 3 10:20 k8s.gpu.amd.com-gpu_c1a10002-0000-4000-8000-000000000002.yaml
Two claims are prepared on this node (gpu-6-133 and gpu-5-132), so there are two per-claim files.
Which device files an AMD GPU needs
Device name gpu-<card>-<render> = the two DRM minor numbers. Every claim also gets /dev/kfd (238:0) and the common env entry. The per-claim CDI name is k8s.gpu.amd.com/gpu=<claim-uid>-<device>.
| File | Major:minor | What it is | Shared? |
|---|---|---|---|
/dev/kfd |
238:0 | KFD (Kernel Fusion Driver), the ROCm compute interface. Queues, memory, events for all GPUs go through it | one file for all GPUs |
/dev/dri/cardN |
226:N | DRM “primary” node of one GPU (display and mode setting heritage; some tools open it) | per GPU |
/dev/dri/renderDN |
226:N (N ≥ 128) | DRM render node of one GPU. ROCm needs it to use that GPU | per GPU |
/dev/kfd alone is not enough. The ROCm runtime enumerates GPUs through KFD but can only use a GPU whose render node the process may open. AMD’s container docs say it directly: pass /dev/kfd plus the individual renderD* nodes to restrict a container to some GPUs; passing all of /dev/dri gives all GPUs. So the isolation boundary is the device cgroup on the render node.
Which files give which GPUs
/dev/kfd /dev/dri/card5 /dev/dri/renderD1321 usable GPU (gpu-5-132). ROCm enumerates GPUs through KFD, but can only use one whose render node the process may open. This is what a Pod with one claimed GPU gets.
/dev/kfdNo usable GPU: the process can enumerate GPUs through KFD, but may open no render node.
/dev/kfd /dev/dri/* (9 card + 8 renderD nodes on this example node: 8 GPUs, plus card0 = the BMC's VGA chip)All 8 GPUs of the node. AMD’s container docs: passing all of /dev/dri gives all GPUs; pass individual renderD* nodes to restrict. A hostPath mount of /dev (as the DRA driver itself has) bypasses DRA entirely.
On the host, /dev/kfd is crw-rw-rw- (mode 0666) and the card/renderD files are owned by groups video and render. Inside a container the cgroup, not the file mode, is what stops access to other GPUs.
From device name to PCI address
The 8 devices on gpu-node-1 are named gpu-1-128, gpu-2-129, … gpu-8-135: the card minor and the render minor of the same GPU. The by-path links tie each pair to a PCI address (example output):
$ ls -l /dev/dri/by-path | head -5
pci-0000:03:00.0-card -> ../card0
pci-0000:11:00.0-card -> ../card1
pci-0000:11:00.0-render -> ../renderD128
pci-0000:2f:00.0-card -> ../card2
pci-0000:2f:00.0-render -> ../renderD129
In this example each MI355X runs as one partition (SPX), and the driver creates one card node and one render node per GPU, so the numbers are contiguous. card0 is a non-AMD display controller (the BMC’s VGA chip) at PCI 03:00.0, with no render node. Do not count on contiguous numbers: some amdgpu versions pre-create spare DRM nodes for compute partitions (XCP, backed by platform devices named amdgpu_xcp_*), and then the minors step by 8. If someone repartitions a GPU, the partition nodes become real devices, and the DRA driver publishes type: amdgpu-partition devices instead. Read device names from the ResourceSlice; never derive them from the GPU index.
Failure modes
- Spec file missing at container create. containerd refuses the container with an “unresolvable CDI devices” error (from the CDI library), and the Pod shows
CreateContainerError. Causes: the driver lost its files, or/var/runwas cleared./var/runis tmpfs, so a reboot wipes all CDI files; that is harmless only because every Pod on the node restarts too and claims are prepared again. - Container restart after a driver upgrade. A restarted container is a new
CreateContainer, so its CDI files must still be there. A new driver version must keep (or rewrite) the files for claims the old version prepared. The AMD driver re-reads its owncheckpoint.jsonfor this; see DRA on the node. - CDI off in the runtime. With
enable_cdi = false(the default before containerd 2.0), CDI IDs are ignored or rejected and the container starts without GPUs. containerd 2.x enables it by default; setting it explicitly in the containerd config is safer. - Device numbering changes. Device nodes are named in the spec by host path and major/minor. If a reboot renumbers cards (for example after a GPU drops off the bus), old spec files are wrong. They are regenerated on the next prepare.
NVIDIA for contrast
NVIDIA also uses CDI, but its specs carry much more than device nodes, because the CUDA user-space driver libraries must match the host kernel driver:
- Device nodes:
/dev/nvidia0…/dev/nvidiaN,/dev/nvidiactl,/dev/nvidia-uvm,/dev/nvidia-uvm-tools, and capability nodes for MIG. - Mounts: host driver libraries such as
libcuda.soandlibnvidia-ml.so, plus binaries likenvidia-smi. - Hooks: an
nvidia-cdi-hookthat updates the linker cache and creates symlinks inside the container.
The NVIDIA Container Toolkit generates a node-wide spec at /var/run/cdi/nvidia.yaml (nvidia-ctk cdi generate, refreshed by the nvidia-cdi-refresh systemd service); device names look like nvidia.com/gpu=0 or nvidia.com/gpu=all. The NVIDIA DRA driver writes its own per-claim specs, like the AMD driver. An AMD container image ships the ROCm user-space itself and needs only the device files, which is why the AMD spec is three device nodes and two env lines.
Look for yourself
# on a GPU node (read-only)
grep -n -E 'enable_cdi|cdi_spec_dirs' /etc/containerd/config.toml
ls -l /var/run/cdi
cat /var/run/cdi/k8s.gpu.amd.com-gpu_common.yaml
ls -l /dev/kfd /dev/dri/by-path