Learn · GPUs in Kubernetes

CDI: how a GPU gets into a container

Updated 10 Nov 2026 · 8 min read

Allocation decides which GPU a Pod gets. Something still has to put that GPU’s device files into the container, and only those. With a DRA driver and containerd 2.x, that something is CDI, the Container Device Interface.

The idea

A container sees only the device files the runtime puts into it. Think of a CDI spec as a recipe card pinned on a board (/var/run/cdi). The card has a name, k8s.gpu.amd.com/gpu=<claim>-gpu-5-132, and says “to cook this, add these three device files and these two environment variables”. The DRA driver writes the card. The kubelet only passes the name along. containerd looks the name up on the board and follows the recipe when it builds the container.

Each part answers one question:

Piece Question it answers
CDI device name Which device does this container want? (a string, nothing more)
Spec file in /var/run/cdi What does that name mean on this node? (device nodes, env, mounts)
containerd’s CDI cache How do I turn names into an OCI spec?
Device cgroup rule Enforce: the container may open these major:minor numbers and no others

Where the analogy breaks: the card is read once, at container create. Nobody re-reads it while the container runs, so deleting the file later does not take the GPU away, and the driver that wrote it is not in the data path. But every container restart is a new create, so the card must still be on the board then.

Before CDI, every vendor shipped its own runtime hook or wrapper (for example nvidia-container-runtime). With CDI the runtime does the work and the vendor only writes data.

Worked example: claim c1a10001 becomes three device files

Follow the prefill Pod …-p4x7k on gpu-node-1, which the scheduler gave gpu-5-132. Every file below is illustrative example output in the driver’s format; the derived values (file name, minor numbers, OCI edits) are computed from the device name at build time and checked against the example file.

From NodePrepareResources to /dev/dri/renderD132

1 · The kubelet asks the driver to prepare the claim

# NodePrepareResources request (gRPC, dra.sock)
claims:
- namespace: gpu-demo
  uid: c1a10001-0000-4000-8000-000000000001
  name: llm-prefill-5f7d8c9b6-p4x7k-gpu-z8k3q

The kubelet sends only the claim’s identity. The driver reads the claim from the API and finds its own result in status.allocation: driver gpu.amd.com, pool gpu-node-1, device gpu-5-132.

2 · The driver writes the CDI spec file

# /var/run/cdi/k8s.gpu.amd.com-gpu_c1a10001-0000-4000-8000-000000000001.yaml
cdiVersion: 0.5.0
kind: k8s.gpu.amd.com/gpu
devices:
    - name: c1a10001-0000-4000-8000-000000000001-gpu-5-132
      containerEdits:
        deviceNodes:
            - path: /dev/kfd
              hostPath: /dev/kfd
              type: c
              major: 238
              permissions: rwm
            - path: /dev/dri/card5
              hostPath: /dev/dri/card5
              type: c
              major: 226
              minor: 5
              permissions: rwm
            - path: /dev/dri/renderD132
              hostPath: /dev/dri/renderD132
              type: c
              major: 226
              minor: 132
              permissions: rwm

File name = kind k8s.gpu.amd.com/gpu with / → -, then _<claim uid>.yaml: /var/run/cdi/k8s.gpu.amd.com-gpu_c1a10001-0000-4000-8000-000000000001.yaml. Device name gpu-5-132 is gpu-<card>-<render>, so the GPU’s nodes are card5 and renderD132, both DRM major 226. /dev/kfd has no minor in the file; the CDI library fills it by stat-ing the host file (238:0).

3 · …and keeps one shared file for every GPU container

# /var/run/cdi/k8s.gpu.amd.com-gpu_common.yaml
cdiVersion: 0.3.0
kind: k8s.gpu.amd.com/gpu
devices:
    - name: common
      containerEdits:
        env:
            - KUBERNETES_NODE_NAME=gpu-node-1
            - DRA_RESOURCE_DRIVER_NAME=gpu.amd.com

Written once when the driver started (1 Nov 08:00). The driver answers the kubelet with two CDI names: k8s.gpu.amd.com/gpu=common and k8s.gpu.amd.com/gpu=c1a10001-0000-4000-8000-000000000001-gpu-5-132.

4 · The kubelet passes the names in CRI CreateContainer

# CRI CreateContainerRequest → config (only the GPU part)
config:
  metadata: {name: prefill}
  CDI_devices:
  - name: k8s.gpu.amd.com/gpu=common
  - name: k8s.gpu.amd.com/gpu=c1a10001-0000-4000-8000-000000000001-gpu-5-132

The kubelet stores the names in its checkpoint and puts them in the CRI field ContainerConfig.CDI_devices every time it creates a container that lists the claim. It never sees device numbers.

5 · containerd resolves the names against its spec dirs

# /etc/containerd/config.toml (excerpt)
version = 3
    enable_cdi = true
    cdi_spec_dirs = ['/etc/cdi', '/var/run/cdi']
Name part Value Found in
kind (vendor/class) k8s.gpu.amd.com/gpu both files
device common 2 env vars k8s.gpu.amd.com-gpu_common.yaml
device c1a10001…-gpu-5-132 3 device nodes k8s.gpu.amd.com-gpu_c1a10001….yaml

/etc/cdi does not exist on GPU nodes; everything is in /var/run/cdi. An unknown name fails the create.

6 · The edits land in the OCI runtime spec

"process": {"env": [ …, "KUBERNETES_NODE_NAME=gpu-node-1", "DRA_RESOURCE_DRIVER_NAME=gpu.amd.com"]},
"linux": {
  "devices": [
    {"path": "/dev/kfd", "type": "c", "major": 238, "minor": 0},
    {"path": "/dev/dri/card5", "type": "c", "major": 226, "minor": 5},
    {"path": "/dev/dri/renderD132", "type": "c", "major": 226, "minor": 132}
  ],
  "resources": {"devices": [
    {"allow": false, "access": "rwm"},          // containerd default: deny all
    {"allow": true, "type": "c", "major": 238, "minor": 0, "access": "rwm"},
    {"allow": true, "type": "c", "major": 226, "minor": 5, "access": "rwm"},
    {"allow": true, "type": "c", "major": 226, "minor": 132, "access": "rwm"}
  ]}
}

For each device node the CDI library adds a linux.devices entry (runc creates the file) and a device-cgroup allow rule (the kernel lets the container open it). Env entries are appended to the process environment.

7 · The container sees one GPU

/dev/kfd             c 238,0    # ROCm compute interface (one for all GPUs)
/dev/dri/card5       c 226,5    # DRM card node of GPU 0000:8b:00.0
/dev/dri/renderD132  c 226,132  # render node of the same GPU
KUBERNETES_NODE_NAME=gpu-node-1
DRA_RESOURCE_DRIVER_NAME=gpu.amd.com

runc creates the container at 10:12:00 (node clock). The cgroup (a BPF device filter on cgroup v2) allows 238:0, 226:5 and 226:132 and nothing else, so the other 7 GPUs’ render nodes cannot be opened even if someone creates them. The model then loads for 4 min 0 s before the Pod turns Ready.

Claim UID c1a10001-0000-4000-8000-000000000001, containerd 2.2, AMD DRA driver v1.0.1 (example).

The same path, hop by hop

Who writes, who reads, who creates
CREATE: EVERY CONTAINER STARTPREPARE: ONCE PER CLAIMkubeletholds the CDI IDsDRA drivergpu-kubeletplugin v1.0.1/var/run/cdicommon + one file perclaimcontainerd 2.2CRI + CDI cacherunccreates the containercontainer prefillkfd, card5, renderD132NodePrepareResourceswrite specCreateContainer + CDIIDsresolve namesOCI specmknod + cgroup
  1. PrepareThe kubelet calls NodePrepareResources; the reply carries the two CDI names.
  2. Write the specDuring prepare the driver writes k8s.gpu.amd.com-gpu_c1a10001….yaml into /var/run/cdi.
  3. CreateThe kubelet sends CRI CreateContainer with CDI_devices = the two names.
  4. Resolvecontainerd’s CDI cache looks the names up in /etc/cdi and /var/run/cdi.
  5. Applycontainerd adds the edits to the OCI spec; runc makes the device nodes and the cgroup rules.
  6. Failure: spec file missingIf the file is gone at create time (driver lost it, or /var/run was wiped), the name does not resolve, containerd refuses the container and the Pod shows CreateContainerError.
  7. Driver gone, container fineOnce the container exists the driver is out of the picture. It keeps its device files even if the driver Pod dies.
Prepare runs once per claim. Create runs every time the container is (re)created.

The spec format

A spec file has cdiVersion (for example 0.5.0), kind (vendor/class), and devices[], each with a name and containerEdits. containerEdits (per device, or top-level for all devices of the kind) holds env, deviceNodes, mounts, hooks, and since 0.7.0 additionalGids and intelRdt. A device’s fully qualified name is <kind>=<name>.

Runtimes look in /etc/cdi (static, written by packages) and /var/run/cdi (dynamic, written at run time). Files in /var/run/cdi win when names collide. On gpu-node-1 there is one common file plus one file per prepared claim, named after the claim UID:

$ ls -l /var/run/cdi          # example output
-rw-------. 1 root root 210 Nov  1 08:00 k8s.gpu.amd.com-gpu_common.yaml
-rw-------. 1 root root 660 Nov  3 10:12 k8s.gpu.amd.com-gpu_c1a10001-0000-4000-8000-000000000001.yaml
-rw-------. 1 root root 660 Nov  3 10:20 k8s.gpu.amd.com-gpu_c1a10002-0000-4000-8000-000000000002.yaml

Two claims are prepared on this node (gpu-6-133 and gpu-5-132), so there are two per-claim files.

Which device files an AMD GPU needs

CDI decoder: one GPU on gpu-node-1
Name parts
PCI address · NUMA node
Held by

Device name gpu-<card>-<render> = the two DRM minor numbers. Every claim also gets /dev/kfd (238:0) and the common env entry. The per-claim CDI name is k8s.gpu.amd.com/gpu=<claim-uid>-<device>.

File Major:minor What it is Shared?
/dev/kfd 238:0 KFD (Kernel Fusion Driver), the ROCm compute interface. Queues, memory, events for all GPUs go through it one file for all GPUs
/dev/dri/cardN 226:N DRM “primary” node of one GPU (display and mode setting heritage; some tools open it) per GPU
/dev/dri/renderDN 226:N (N ≥ 128) DRM render node of one GPU. ROCm needs it to use that GPU per GPU

/dev/kfd alone is not enough. The ROCm runtime enumerates GPUs through KFD but can only use a GPU whose render node the process may open. AMD’s container docs say it directly: pass /dev/kfd plus the individual renderD* nodes to restrict a container to some GPUs; passing all of /dev/dri gives all GPUs. So the isolation boundary is the device cgroup on the render node.

Which files give which GPUs

Container gets
/dev/kfd  /dev/dri/card5  /dev/dri/renderD132

1 usable GPU (gpu-5-132). ROCm enumerates GPUs through KFD, but can only use one whose render node the process may open. This is what a Pod with one claimed GPU gets.

On the host, /dev/kfd is crw-rw-rw- (mode 0666) and the card/renderD files are owned by groups video and render. Inside a container the cgroup, not the file mode, is what stops access to other GPUs.

From device name to PCI address

The 8 devices on gpu-node-1 are named gpu-1-128, gpu-2-129, … gpu-8-135: the card minor and the render minor of the same GPU. The by-path links tie each pair to a PCI address (example output):

$ ls -l /dev/dri/by-path | head -5
pci-0000:03:00.0-card   -> ../card0
pci-0000:11:00.0-card   -> ../card1
pci-0000:11:00.0-render -> ../renderD128
pci-0000:2f:00.0-card   -> ../card2
pci-0000:2f:00.0-render -> ../renderD129

In this example each MI355X runs as one partition (SPX), and the driver creates one card node and one render node per GPU, so the numbers are contiguous. card0 is a non-AMD display controller (the BMC’s VGA chip) at PCI 03:00.0, with no render node. Do not count on contiguous numbers: some amdgpu versions pre-create spare DRM nodes for compute partitions (XCP, backed by platform devices named amdgpu_xcp_*), and then the minors step by 8. If someone repartitions a GPU, the partition nodes become real devices, and the DRA driver publishes type: amdgpu-partition devices instead. Read device names from the ResourceSlice; never derive them from the GPU index.

Failure modes

  • Spec file missing at container create. containerd refuses the container with an “unresolvable CDI devices” error (from the CDI library), and the Pod shows CreateContainerError. Causes: the driver lost its files, or /var/run was cleared. /var/run is tmpfs, so a reboot wipes all CDI files; that is harmless only because every Pod on the node restarts too and claims are prepared again.
  • Container restart after a driver upgrade. A restarted container is a new CreateContainer, so its CDI files must still be there. A new driver version must keep (or rewrite) the files for claims the old version prepared. The AMD driver re-reads its own checkpoint.json for this; see DRA on the node.
  • CDI off in the runtime. With enable_cdi = false (the default before containerd 2.0), CDI IDs are ignored or rejected and the container starts without GPUs. containerd 2.x enables it by default; setting it explicitly in the containerd config is safer.
  • Device numbering changes. Device nodes are named in the spec by host path and major/minor. If a reboot renumbers cards (for example after a GPU drops off the bus), old spec files are wrong. They are regenerated on the next prepare.

NVIDIA for contrast

NVIDIA also uses CDI, but its specs carry much more than device nodes, because the CUDA user-space driver libraries must match the host kernel driver:

  • Device nodes: /dev/nvidia0 … /dev/nvidiaN, /dev/nvidiactl, /dev/nvidia-uvm, /dev/nvidia-uvm-tools, and capability nodes for MIG.
  • Mounts: host driver libraries such as libcuda.so and libnvidia-ml.so, plus binaries like nvidia-smi.
  • Hooks: an nvidia-cdi-hook that updates the linker cache and creates symlinks inside the container.

The NVIDIA Container Toolkit generates a node-wide spec at /var/run/cdi/nvidia.yaml (nvidia-ctk cdi generate, refreshed by the nvidia-cdi-refresh systemd service); device names look like nvidia.com/gpu=0 or nvidia.com/gpu=all. The NVIDIA DRA driver writes its own per-claim specs, like the AMD driver. An AMD container image ships the ROCm user-space itself and needs only the device files, which is why the AMD spec is three device nodes and two env lines.

Look for yourself

# on a GPU node (read-only)
grep -n -E 'enable_cdi|cdi_spec_dirs' /etc/containerd/config.toml
ls -l /var/run/cdi
cat /var/run/cdi/k8s.gpu.amd.com-gpu_common.yaml
ls -l /dev/kfd /dev/dri/by-path

Further reading