Learn · Overview
GPUs in Kubernetes (overview)
The idea
Kubernetes has no idea what a GPU is. Four separate pieces of software teach it, and each one fails in its own way:
- an installer (the AMD GPU operator) that puts the GPU agents on every GPU node;
- a booking system (DRA) that knows every GPU by name and gives each Pod one, like a theatre seat;
- a fitter (the DRA driver on the node, plus CDI in containerd) that makes the booked GPU’s device files appear inside the container, and nothing else;
- a cabler (Multus plus the RDMA device plugin) that gives the Pod a second network interface and RDMA access to a 400G NIC.
If you know Kubernetes well, the short version is: GPUs are allocated like PersistentVolumeClaims (DRA), injected like CSI volumes (a kubelet plugin over gRPC, then CDI), installed by an operator, and networked through Multus plus a device plugin. The analogy with storage breaks in one place: there is no external provisioner. The scheduler itself picks the device.
- GPU nodes in the sample cluster
- 4
- MI355X GPUs
- 32
- 8 per node, whole GPUs
- GPUs in claims
- 16
- at snapshot time
- Operator DaemonSets
- 3
- DRA driver, labeller, exporter
One Pod through every layer
The whole track follows one sample Pod, …-p4x7k of llm-prefill on gpu-node-1. Press play to see who talks to whom, then step through what each layer leaves behind.
What each layer leaves behind for Pod …-p4x7k
1 · Install: the operator renders DaemonSets
$ kubectl -n kube-amd-gpu get ds
NAME DESIRED READY NODE SELECTOR
default-dra-driver 4 4 example.com/gpu-vendor=AMD
default-metrics-exporter 4 4 example.com/gpu-vendor=AMD
default-node-labeller 4 4 example.com/gpu-vendor=AMDThe cluster’s deployment tooling installs the AMD GPU operator chart. The controller reads DeviceConfig kube-amd-gpu/default and renders three DaemonSets on the 4 AMD GPU nodes. Here the host driver (amdgpu-dkms) is installed with the OS image, not by the operator. → GPU operators
2 · Publish: 8 GPUs in a ResourceSlice
gpu-1-128 pciBusID 0000:11:00.0 numaNode 0 memory 294912Mi
gpu-2-129 pciBusID 0000:2f:00.0 numaNode 0 memory 294912Mi
gpu-3-130 pciBusID 0000:4d:00.0 numaNode 0 memory 294912Mi
gpu-4-131 pciBusID 0000:6b:00.0 numaNode 0 memory 294912Mi
gpu-5-132 pciBusID 0000:8b:00.0 numaNode 1 memory 294912Mi
gpu-6-133 pciBusID 0000:a9:00.0 numaNode 1 memory 294912Mi
gpu-7-134 pciBusID 0000:c7:00.0 numaNode 1 memory 294912Mi
gpu-8-135 pciBusID 0000:e5:00.0 numaNode 1 memory 294912MiThe DRA driver finds the GPUs in sysfs, registers gpu.amd.com with the kubelet and publishes one ResourceSlice for the node, with typed attributes and capacities. → DRA objects
3 · Allocate: the claim names gpu-5-132
status:
allocation:
devices:
results:
- request: gpu
driver: gpu.amd.com
pool: gpu-node-1
device: gpu-5-132
nodeSelector:
nodeSelectorTerms:
- matchFields:
- {key: metadata.name, operator: In, values: [gpu-node-1]}
reservedFor:
- resource: pods
name: llm-prefill-5f7d8c9b6-p4x7k
uid: b0d10001-0000-4000-8000-000000000001The resourceclaim controller stamps a claim out of the Deployment’s template; the scheduler picks a free device and writes (gpu.amd.com, gpu-node-1, gpu-5-132) into it before binding the Pod. → GPU allocation
4 · Prepare: the driver answers with CDI IDs
"preparedClaims": {
"c1a10001-0000-4000-8000-000000000001": [{
"request_names": ["gpu"],
"pool_name": "gpu-node-1",
"device_name": "gpu-5-132",
"cdi_device_ids": [
"k8s.gpu.amd.com/gpu=common",
"k8s.gpu.amd.com/gpu=c1a10001-0000-4000-8000-000000000001-gpu-5-132"
]
}]
}The kubelet calls NodePrepareResources on the driver’s socket. The driver writes a CDI spec and its checkpoint, and returns two CDI device IDs. → DRA on the node
5 · Inject: three device nodes in the container
/dev/kfd c 238,0 # ROCm compute interface (one for all GPUs)
/dev/dri/card5 c 226,5 # DRM card node of GPU 0000:8b:00.0
/dev/dri/renderD132 c 226,132 # render node of the same GPU
KUBERNETES_NODE_NAME=gpu-node-1
DRA_RESOURCE_DRIVER_NAME=gpu.amd.comcontainerd resolves the CDI IDs against /var/run/cdi and adds exactly these device nodes, with a device cgroup that allows only them. → CDI
6 · Network: eth0 for Kubernetes, net1 for data
[
{
"name": "cilium",
"interface": "eth0",
"ips": [
"<pod-ip>"
],
"mac": "02:00:00:…",
"default": true,
"gateway": [
"<gw>"
]
},
{
"name": "default/multus-data-net-5",
"interface": "net1",
"ips": [
"<data-net-ip>"
],
"mac": "02:00:01:…"
}
]Multus calls Cilium for eth0, then adds net1 from the NAD default/multus-data-net-5 (macvlan on a 400G NIC, address from whereabouts). The RDMA shared device plugin hands over the RDMA device files for rdma/hca_devices: 1. GPU gpu-5-132 (GPU index 4) shares a PCIe switch with NIC ens5np0 (rdma4), both on NUMA node 1. → Pod networking
7 · Ready: 4 min 0 s later
API 10:00:00 Deployment and ReplicaSet created, replicas: 12
API 10:12:00 ReplicaSet controller creates the Pod; resourceclaim controller creates its claim
API 10:12:00 Scheduler allocates gpu-5-132, binds the Pod (PodScheduled=True)
node 10:11:58.900 containerd RunPodSandbox; Multus adds eth0 and net1
node 10:11:59 init container write-config runs, exits 0
node 10:12:00 container prefill started (CDI edits applied)
node 10:12:01.100 kubelet logs podStartE2EDuration="1.1s"
API 10:16:00 Ready=True: model loaded, readiness probe passesKubernetes was done within seconds of the Pod’s creation; the rest until Ready is model loading. A claim in state allocated,reserved says the GPU is assigned, not that the model is loaded.
Illustrative example objects and files from the sample cluster, trimmed. Each step links to the page that explains it.
Where it breaks
Pick a symptom: the diagram shows where the flow stops, and the first thing to check.
- Free devices on nodes the Pod may use; selector typos (a missing attribute is an error, not false).
- RDMA shared device plugin Pod on the node; hca_devices.sock in /var/lib/kubelet/device-plugins.
- DRA driver Pod on that node; kubelet log for NodePrepareResources.
- The claim's spec file in /var/run/cdi; containerd enable_cdi and cdi_spec_dirs.
- NAD exists; per-node CNI file in /etc/cni/multus/net.d; IP pool not exhausted.
- DRA driver Pod stuck in its driver-init container (waits for /sys/class/kfd).
- DRA driver down on that node: the kubelet cannot call NodeUnprepareResources.
Which object answers which question
| Question | Look at |
|---|---|
| Which GPUs exist on a node, and what are they? | kubectl get resourceslices --field-selector spec.nodeName=<node> -o yaml |
| Which Pod holds which GPU? | kubectl get resourceclaims -A -o wide, then status.allocation.devices.results |
| Is the driver registered on the node? | /var/lib/kubelet/plugins_registry/gpu.amd.com-reg.sock; a ResourceSlice exists for the node |
| Did the device files reach the container? | /var/run/cdi/*_<claim-uid>.yaml on the node; the driver’s checkpoint.json |
| Which operands are on and at which version? | kubectl -n kube-amd-gpu get deviceconfig default -o yaml, get ds -o wide |
| Does the Pod have its data network? | Pod annotation k8s.v1.cni.cncf.io/network-status |
| Is the node fit for new GPU Pods? | Ready condition, no NoSchedule taints, a ResourceSlice with all GPUs |
All symptoms at a glance
| Symptom | Layer | First check | Page |
|---|---|---|---|
Pod Pending, event mentions claims or CEL |
allocation | free devices on allowed nodes; selector typos | DRA objects |
Pod Pending, Insufficient rdma/hca_devices |
device plugin | RDMA plugin registered on the node | Pod networking |
Pod ContainerCreating, prepare errors |
node DRA | DRA driver Pod on that node; kubelet log | DRA on the node |
CreateContainerError, unresolvable CDI device |
CDI | spec file in /var/run/cdi |
CDI |
FailedCreatePodSandBox |
Multus / whereabouts | NAD exists; per-node CNI file; IP pool | Pod networking |
| Node has no ResourceSlice | operator / driver | DRA driver Pod stuck in Init (amdgpu not loaded) |
GPU operators |
GPU Pods stuck Terminating |
node DRA | driver down, cannot unprepare | DRA on the node |
| Node tainted not-ready | host readiness | which check failed in the node’s conditions | kubectl describe node |
Quick commands
kubectl get nodes -l example.com/gpu-node -o wide
kubectl -n kube-amd-gpu get deviceconfig,ds,deploy
kubectl get deviceclass,resourceslices
kubectl get resourceclaims -A -o wide
kubectl get pods -A -o wide --field-selector spec.nodeName=<gpu-node>
kubectl get net-attach-def -A
The sample cluster’s choices, in one place
- DRA only, device plugin off; whole GPUs, SPX/NPS1, one claim per Pod.
- Host driver and ROCm in the OS image (
amdgpu-dkms), not from the operator (driver.enable: false); NFD, KMM, partitioning, remediation and test runner off. - Readiness through node conditions and a
NoScheduletaint while a check fails, with DRA driver registration as one of the checks. - Data network through file-backed Multus NADs, one per NIC; RDMA through one shared resource that exposes all 8 NICs.
AMD versus NVIDIA
| AMD (MI355X nodes) | NVIDIA | |
|---|---|---|
| Operator resource | DeviceConfig |
ClusterPolicy (device plugin) or GPUCluster (DRA), never both |
| DRA driver name | gpu.amd.com |
gpu.nvidia.com, mig.nvidia.com, plus ComputeDomain for NVLink across trays |
| Kernel driver | host RPM (DKMS), or the operator’s driver install | host or driver container |
| CDI content | 3 device nodes, 2 env vars | device nodes, host driver libraries, hooks |
| RDMA stack | in-box mlx5, RDMA shared device plugin |
Network Operator (NicClusterPolicy) |