Learn · Overview

GPUs in Kubernetes (overview)

Updated 10 Nov 2026 · 4 min read

What this overview is built from
Kernels & siliconGPU fundamentalsGPUs in KubernetesBottom row: the smallest building blocks. Each line goes down from a page to a page it is built from; overviews (bold border) sit on top. Hover a box to light up what it is built from and what it is part of; click to open it.

The idea

Kubernetes has no idea what a GPU is. Four separate pieces of software teach it, and each one fails in its own way:

  • an installer (the AMD GPU operator) that puts the GPU agents on every GPU node;
  • a booking system (DRA) that knows every GPU by name and gives each Pod one, like a theatre seat;
  • a fitter (the DRA driver on the node, plus CDI in containerd) that makes the booked GPU’s device files appear inside the container, and nothing else;
  • a cabler (Multus plus the RDMA device plugin) that gives the Pod a second network interface and RDMA access to a 400G NIC.

If you know Kubernetes well, the short version is: GPUs are allocated like PersistentVolumeClaims (DRA), injected like CSI volumes (a kubelet plugin over gRPC, then CDI), installed by an operator, and networked through Multus plus a device plugin. The analogy with storage breaks in one place: there is no external provisioner. The scheduler itself picks the device.

GPU nodes in the sample cluster
4
MI355X GPUs
32
8 per node, whole GPUs
GPUs in claims
16
at snapshot time
Operator DaemonSets
3
DRA driver, labeller, exporter

One Pod through every layer

The whole track follows one sample Pod, …-p4x7k of llm-prefill on gpu-node-1. Press play to see who talks to whom, then step through what each layer leaves behind.

From a physical GPU to a running prefill container
STARTINSTALLALLOCATEGPU operatorDRA driverAPI + schedulerkubelet + containerdprefill PodDeviceConfig → DaemonSet ongpu-vendor=AMD nodesregister gpu.amd.comResourceSlice: 8 GPUs +attributesclaim from template; schedulerallocates gpu-5-132NodePrepareResources → CDIIDsMultus: eth0 + net1; RDMAplugin: /dev/infiniband/*container with /dev/kfd, card5,renderD132
Illustrative names from the sample cluster node gpu-node-1.

What each layer leaves behind for Pod …-p4x7k

1 · Install: the operator renders DaemonSets

$ kubectl -n kube-amd-gpu get ds
NAME                       DESIRED  READY  NODE SELECTOR
default-dra-driver         4        4      example.com/gpu-vendor=AMD
default-metrics-exporter   4        4      example.com/gpu-vendor=AMD
default-node-labeller      4        4      example.com/gpu-vendor=AMD

The cluster’s deployment tooling installs the AMD GPU operator chart. The controller reads DeviceConfig kube-amd-gpu/default and renders three DaemonSets on the 4 AMD GPU nodes. Here the host driver (amdgpu-dkms) is installed with the OS image, not by the operator. → GPU operators

2 · Publish: 8 GPUs in a ResourceSlice

gpu-1-128   pciBusID 0000:11:00.0  numaNode 0  memory 294912Mi
gpu-2-129   pciBusID 0000:2f:00.0  numaNode 0  memory 294912Mi
gpu-3-130   pciBusID 0000:4d:00.0  numaNode 0  memory 294912Mi
gpu-4-131   pciBusID 0000:6b:00.0  numaNode 0  memory 294912Mi
gpu-5-132   pciBusID 0000:8b:00.0  numaNode 1  memory 294912Mi
gpu-6-133   pciBusID 0000:a9:00.0  numaNode 1  memory 294912Mi
gpu-7-134   pciBusID 0000:c7:00.0  numaNode 1  memory 294912Mi
gpu-8-135   pciBusID 0000:e5:00.0  numaNode 1  memory 294912Mi

The DRA driver finds the GPUs in sysfs, registers gpu.amd.com with the kubelet and publishes one ResourceSlice for the node, with typed attributes and capacities. → DRA objects

3 · Allocate: the claim names gpu-5-132

status:
  allocation:
    devices:
      results:
      - request: gpu
        driver: gpu.amd.com
        pool: gpu-node-1
        device: gpu-5-132
    nodeSelector:
      nodeSelectorTerms:
      - matchFields:
        - {key: metadata.name, operator: In, values: [gpu-node-1]}
  reservedFor:
  - resource: pods
    name: llm-prefill-5f7d8c9b6-p4x7k
    uid: b0d10001-0000-4000-8000-000000000001

The resourceclaim controller stamps a claim out of the Deployment’s template; the scheduler picks a free device and writes (gpu.amd.com, gpu-node-1, gpu-5-132) into it before binding the Pod. → GPU allocation

4 · Prepare: the driver answers with CDI IDs

"preparedClaims": {
  "c1a10001-0000-4000-8000-000000000001": [{
    "request_names": ["gpu"],
    "pool_name": "gpu-node-1",
    "device_name": "gpu-5-132",
    "cdi_device_ids": [
      "k8s.gpu.amd.com/gpu=common",
      "k8s.gpu.amd.com/gpu=c1a10001-0000-4000-8000-000000000001-gpu-5-132"
    ]
  }]
}

The kubelet calls NodePrepareResources on the driver’s socket. The driver writes a CDI spec and its checkpoint, and returns two CDI device IDs. → DRA on the node

5 · Inject: three device nodes in the container

/dev/kfd             c 238,0    # ROCm compute interface (one for all GPUs)
/dev/dri/card5       c 226,5    # DRM card node of GPU 0000:8b:00.0
/dev/dri/renderD132  c 226,132  # render node of the same GPU
KUBERNETES_NODE_NAME=gpu-node-1
DRA_RESOURCE_DRIVER_NAME=gpu.amd.com

containerd resolves the CDI IDs against /var/run/cdi and adds exactly these device nodes, with a device cgroup that allows only them. → CDI

6 · Network: eth0 for Kubernetes, net1 for data

[
  {
    "name": "cilium",
    "interface": "eth0",
    "ips": [
      "<pod-ip>"
    ],
    "mac": "02:00:00:…",
    "default": true,
    "gateway": [
      "<gw>"
    ]
  },
  {
    "name": "default/multus-data-net-5",
    "interface": "net1",
    "ips": [
      "<data-net-ip>"
    ],
    "mac": "02:00:01:…"
  }
]

Multus calls Cilium for eth0, then adds net1 from the NAD default/multus-data-net-5 (macvlan on a 400G NIC, address from whereabouts). The RDMA shared device plugin hands over the RDMA device files for rdma/hca_devices: 1. GPU gpu-5-132 (GPU index 4) shares a PCIe switch with NIC ens5np0 (rdma4), both on NUMA node 1. → Pod networking

7 · Ready: 4 min 0 s later

API  10:00:00      Deployment and ReplicaSet created, replicas: 12
API  10:12:00      ReplicaSet controller creates the Pod; resourceclaim controller creates its claim
API  10:12:00      Scheduler allocates gpu-5-132, binds the Pod (PodScheduled=True)
node 10:11:58.900  containerd RunPodSandbox; Multus adds eth0 and net1
node 10:11:59      init container write-config runs, exits 0
node 10:12:00      container prefill started (CDI edits applied)
node 10:12:01.100  kubelet logs podStartE2EDuration="1.1s"
API  10:16:00      Ready=True: model loaded, readiness probe passes

Kubernetes was done within seconds of the Pod’s creation; the rest until Ready is model loading. A claim in state allocated,reserved says the GPU is assigned, not that the model is loaded.

Illustrative example objects and files from the sample cluster, trimmed. Each step links to the page that explains it.

Where it breaks

Pick a symptom: the diagram shows where the flow stops, and the first thing to check.

Symptom
ContainerCreating: prepare error: the node DRA layer
GPU NODEAMD GPU operatorDeviceConfig →DaemonSetsAPI + schedulerslices, claims, bindingDRA drivergpu.amd.comkubeletRDMA device pluginrdma/hca_devicesMultus +whereaboutsnet1containerdCDIprefill containerDaemonSetResourceSlicebound PodprepareAllocateCNI ADDCreateContainerdevice nodes
  1. DRA driver Pod on that node; kubelet log for NodePrepareResources.

Which object answers which question

Question Look at
Which GPUs exist on a node, and what are they? kubectl get resourceslices --field-selector spec.nodeName=<node> -o yaml
Which Pod holds which GPU? kubectl get resourceclaims -A -o wide, then status.allocation.devices.results
Is the driver registered on the node? /var/lib/kubelet/plugins_registry/gpu.amd.com-reg.sock; a ResourceSlice exists for the node
Did the device files reach the container? /var/run/cdi/*_<claim-uid>.yaml on the node; the driver’s checkpoint.json
Which operands are on and at which version? kubectl -n kube-amd-gpu get deviceconfig default -o yaml, get ds -o wide
Does the Pod have its data network? Pod annotation k8s.v1.cni.cncf.io/network-status
Is the node fit for new GPU Pods? Ready condition, no NoSchedule taints, a ResourceSlice with all GPUs

All symptoms at a glance

Symptom Layer First check Page
Pod Pending, event mentions claims or CEL allocation free devices on allowed nodes; selector typos DRA objects
Pod Pending, Insufficient rdma/hca_devices device plugin RDMA plugin registered on the node Pod networking
Pod ContainerCreating, prepare errors node DRA DRA driver Pod on that node; kubelet log DRA on the node
CreateContainerError, unresolvable CDI device CDI spec file in /var/run/cdi CDI
FailedCreatePodSandBox Multus / whereabouts NAD exists; per-node CNI file; IP pool Pod networking
Node has no ResourceSlice operator / driver DRA driver Pod stuck in Init (amdgpu not loaded) GPU operators
GPU Pods stuck Terminating node DRA driver down, cannot unprepare DRA on the node
Node tainted not-ready host readiness which check failed in the node’s conditions kubectl describe node

Quick commands

kubectl get nodes -l example.com/gpu-node -o wide
kubectl -n kube-amd-gpu get deviceconfig,ds,deploy
kubectl get deviceclass,resourceslices
kubectl get resourceclaims -A -o wide
kubectl get pods -A -o wide --field-selector spec.nodeName=<gpu-node>
kubectl get net-attach-def -A

The sample cluster’s choices, in one place

  • DRA only, device plugin off; whole GPUs, SPX/NPS1, one claim per Pod.
  • Host driver and ROCm in the OS image (amdgpu-dkms), not from the operator (driver.enable: false); NFD, KMM, partitioning, remediation and test runner off.
  • Readiness through node conditions and a NoSchedule taint while a check fails, with DRA driver registration as one of the checks.
  • Data network through file-backed Multus NADs, one per NIC; RDMA through one shared resource that exposes all 8 NICs.

AMD versus NVIDIA

AMD (MI355X nodes) NVIDIA
Operator resource DeviceConfig ClusterPolicy (device plugin) or GPUCluster (DRA), never both
DRA driver name gpu.amd.com gpu.nvidia.com, mig.nvidia.com, plus ComputeDomain for NVLink across trays
Kernel driver host RPM (DKMS), or the operator’s driver install host or driver container
CDI content 3 device nodes, 2 env vars device nodes, host driver libraries, hooks
RDMA stack in-box mlx5, RDMA shared device plugin Network Operator (NicClusterPolicy)

Further reading