Learn · GPUs in Kubernetes
DRA on the node: registration, prepare, checkpoints and restarts
The scheduler only records an allocation in a ResourceClaim. On the node, the kubelet and the vendor’s kubelet plugin turn that record into device files in a container. This page follows that path on gpu-node-1, with the AMD DRA driver v1.0.1 and kubelet 1.35 (illustrative example output).
The idea
Think of the kubelet as a hotel front desk and the DRA driver as the GPU specialist on call. The specialist leaves a phone number in the desk’s directory (a socket in plugins_registry). When a guest (a Pod) arrives with a booking (an allocated claim), the desk calls the specialist: “prepare room gpu-5-132 for claim c1a10001”. The specialist sets it up and hands back a key card (the CDI names). Both keep a ledger of who is in which room, so either can be restarted without forgetting.
| Piece | Question it answers |
|---|---|
| Registration socket | How does the kubelet find the driver at all? |
| ResourceSlice | What GPUs exist on this node? (published by the driver) |
NodePrepareResources |
Make this claim’s device usable; what should the container get? |
| Two checkpoints | After a restart, which claims are already prepared? |
| 30 s wiping delay | How long does the kubelet trust a driver that went away? |
Where the analogy breaks: once the container is running, the specialist is not needed. The device files stay even if the driver dies. The driver matters only when a container is created or a claim is released.
- Devices published
- 8
- one ResourceSlice for the node
- Claims prepared now
- 2
- gpu-6-133, gpu-5-132
- Driver Pod start
- ~1 s
- example kubelet log
- Prepare timeout
- 45 s
- kubelet default per call
- Wiping delay
- 30 s
- kubelet 1.35 defaultWipingDelay
Worked example: the kubelet prepares claim c1a10001
The same prefill Pod as on GPU allocation and CDI. Files and log lines are illustrative example output in the driver’s format; the two gRPC messages are rebuilt from the checkpoints (same fields as api.proto), since the wire messages are not logged.
From driver start to a prepared GPU
1 · 1 Nov 08:00: the driver finds 8 GPUs
I1101 08:00:00.100000 main.go:151] Feature gates: ContextualLogging=true
I1101 08:00:00.200000 discovery.go:136] Found full AMD GPU: gpu-3-130, compute type: spx, memory type: nps1
I1101 08:00:00.200100 discovery.go:136] Found full AMD GPU: gpu-4-131, compute type: spx, memory type: nps1
I1101 08:00:00.200200 discovery.go:136] Found full AMD GPU: gpu-5-132, compute type: spx, memory type: nps1
…
I1101 08:00:00.201000 discovery.go:177] Discovered 8 AMD GPU devices
I1101 08:00:00.300000 driver.go:106] Publishing ResourceSlice: …The init container waited for amdgpu; then gpu-kubeletplugin read sysfs. Each GPU is named gpu-<card>-<render> and is a whole GPU (SPX compute, NPS1 memory).
2 · It opens two sockets
$ ls -la /var/lib/kubelet/plugins_registry /var/lib/kubelet/plugins/gpu.amd.com
srwxr-xr-x. 1 root root 0 Nov 1 08:00 gpu.amd.com-reg.sock # kubelet watches this dir
srwxr-xr-x. 1 root root 0 Nov 1 08:00 dra.sock # the gRPC service
-rw-------. 1 root root 1200 Nov 3 10:13 checkpoint.json # driver's prepared claimsdra.sock is the service. gpu.amd.com-reg.sock in plugins_registry is the “phone number”: the kubelet’s plugin watcher (the same mechanism CSI uses) sees it appear. Socket times = last driver start.
3 · The kubelet calls GetInfo and registers the plugin
# kubelet → gpu.amd.com-reg.sock: GetInfo() (PluginInfo fields; values reconstructed)
type: DRAPlugin
name: gpu.amd.com
endpoint: /var/lib/kubelet/plugins/gpu.amd.com/dra.sock
supported_versions: [v1.DRAPlugin, v1beta1.DRAPlugin] # kubelet 1.35 picks v1The reply names the driver, the endpoint and the gRPC services, as <package>.<service>. The kubelet picks one it supports. From now on any claim with driver gpu.amd.com on this node goes to dra.sock.
4 · The driver publishes the ResourceSlice
# ResourceSlice 00000-gpu.amd.com-gpu-node-1-x7q2m (1 of 8 devices)
- name: gpu-5-132
attributes:
productName: {string: Instinct_MI355X_OAM}
partitionProfile: {string: spx_nps1}
numaNode: {int: 1}
resource.kubernetes.io/pciBusID: {string: "0000:8b:00.0"}
capacity:
memory: {value: 294912Mi}
computeUnits: {value: "256"}Written through the API server with the driver’s ServiceAccount amd-gpu-operator-dra-driver. The scheduler reads this; it picked gpu-5-132 (PCI 0000:8b:00.0, NUMA 1) for the claim.
5 · 3 Nov: the Pod is bound; the kubelet writes its ledger first
# /var/lib/kubelet/dra_manager_state (kind DRACheckpoint), this claim only
ClaimUID: c1a10001-0000-4000-8000-000000000001
ClaimName: llm-prefill-5f7d8c9b6-p4x7k-gpu-z8k3q
Namespace: gpu-demo
PodUIDs: {b0d10001-0000-4000-8000-000000000001: {}}
DriverState: {} # filled in after the driver answers (step 6)The kubelet checks the claim is allocated and reserved for this Pod, then records it before calling the driver, so a crash in between does not lose track of it.
6 · NodePrepareResources: request
# NodePrepareResourcesRequest → dra.sock
claims:
- namespace: gpu-demo
uid: c1a10001-0000-4000-8000-000000000001
name: llm-prefill-5f7d8c9b6-p4x7k-gpu-z8k3qOnly the claim’s identity travels; the driver fetches the claim and reads status.allocation for its own driver name. Calls are batched per Pod and, through the kubeletplugin helper, serialized: two prepares never run at once. Timeout 45 s.
7 · The driver writes the CDI file and its own ledger, then answers
"preparedClaims": {
"c1a10001-0000-4000-8000-000000000001": [{
"request_names": ["gpu"],
"pool_name": "gpu-node-1",
"device_name": "gpu-5-132",
"cdi_device_ids": [
"k8s.gpu.amd.com/gpu=common",
"k8s.gpu.amd.com/gpu=c1a10001-0000-4000-8000-000000000001-gpu-5-132"
]
}]
}# NodePrepareResourcesResponse ← dra.sock
claims:
c1a10001-0000-4000-8000-000000000001: # keyed by claim UID
devices:
- request_names: [gpu]
pool_name: gpu-node-1
device_name: gpu-5-132
cdi_device_ids:
- k8s.gpu.amd.com/gpu=common
- k8s.gpu.amd.com/gpu=c1a10001-0000-4000-8000-000000000001-gpu-5-132
error: ""The CDI file is k8s.gpu.amd.com-gpu_c1a10001-0000-4000-8000-000000000001.yaml (what is in it). Prepare is idempotent: asked again (say after a kubelet restart), the driver finds the UID in checkpoint.json and returns the same answer.
8 · The kubelet stores the CDI names and creates the container
DriverState:
gpu.amd.com:
Devices:
- PoolName: gpu-node-1
DeviceName: gpu-5-132
RequestNames: [gpu]
CDIDeviceIDs: [k8s.gpu.amd.com/gpu=common, k8s.gpu.amd.com/gpu=c1a10001…-gpu-5-132]The kubelet passes the two names to containerd in CreateContainer for every container that lists the claim. The container started at 10:12:00 (node clock); the model took 4 min 0 s more to load.
9 · Health check: four records agree
| Record | Where | Claims for gpu-node-1 |
|---|---|---|
| API: claims with pool = node | kubectl get resourceclaims -A | 2 |
| kubelet: dra_manager_state | ClaimInfoStateList | 2 |
| driver: checkpoint.json | v1.preparedClaims | 2 |
| runtime: /var/run/cdi | per-claim files | 2 |
All four list the same 2 claims (gpu-6-133, gpu-5-132); the other 6 GPUs are free. A mismatch means a leak (a CDI file or checkpoint entry with no claim) or a lost prepare. Commands are under Look for yourself.
gpu-node-1, AMD DRA driver v1.0.1, kubelet 1.35. Driver start 1 Nov; Pod start 3 Nov 10:12 (node clock). Example output.
The whole conversation, including unprepare, as one sequence:
Prepare fails (driver down, timeout, error): the Pod stays in ContainerCreating with an event, and the kubelet retries. Nothing is rescheduled: the claim is already allocated to this node. The helper documentation also asks the driver to be the last line of defence and refuse a device already prepared for another claim.
Unprepare runs when the last Pod on the node that uses a claim has stopped. The driver deletes the CDI file and the checkpoint.json entry; the kubelet drops its record, and the resourceclaim controller frees the allocation in the API. A kubelet reconcile loop (every 60 s) retries unprepare for claims whose Pods are gone. Until unprepare succeeds the Pod is not reported terminated, so the GPU cannot be given to someone else.
Both checkpoint files carry a checksum, so a hand-edited or truncated file fails the check.
The driver Pod
The AMD GPU operator renders the DRA driver as DaemonSet kube-amd-gpu/default-dra-driver. The parts that matter (example, trimmed):
spec:
updateStrategy:
rollingUpdate: {maxSurge: 0, maxUnavailable: 1} # old Pod stops before new starts
template:
spec:
priorityClassName: system-node-critical
nodeSelector: {example.com/gpu-vendor: AMD}
tolerations: [{operator: Exists}] # runs even on tainted nodes
initContainers:
- name: driver-init # wait for the amdgpu module
command: [sh, -c, 'while [ ! -d /sys/class/kfd ] || [ ! -d /sys/module/amdgpu/drivers/ ]; do echo "amdgpu driver is not loaded "; sleep 2; done']
containers:
- name: plugin
image: registry.local/rocm/k8s-gpu-dra-driver:v1.0.1
command: [gpu-kubeletplugin]
env:
- {name: CDI_ROOT, value: /var/run/cdi}
- {name: KUBELET_REGISTRAR_DIRECTORY_PATH, value: /var/lib/kubelet/plugins_registry}
- {name: KUBELET_PLUGINS_DIRECTORY_PATH, value: /var/lib/kubelet/plugins}
- name: NODE_NAME
valueFrom: {fieldRef: {fieldPath: spec.nodeName}}
securityContext: {privileged: true}
volumeMounts: # all hostPath
- {name: plugins-registry, mountPath: /var/lib/kubelet/plugins_registry}
- {name: plugins, mountPath: /var/lib/kubelet/plugins}
- {name: cdi, mountPath: /var/run/cdi}
- {name: dev, mountPath: /dev}
- {name: sys, mountPath: /sys}
Read it as a list of facts:
- It needs the host driver first. The init container loops until
/sys/class/kfdand/sys/module/amdgpu/drivers/exist. A node with a brokenamdgpuinstall shows the driver Pod stuck inInit:0/1and no ResourceSlice. - It runs everywhere it is selected.
operator: Existstolerates every taint, so tainting a GPU node (for example with a not-ready taint) does not stop the DRA driver. That is deliberate: Pods that are already running must still be unprepared when they end. - It is privileged and mounts
/devand/sysfrom the host, because it reads GPU topology from sysfs. - No liveness probe. The driver has a gRPC health server, but its
--healthcheck-portdefaults to-1(off) and the DaemonSet sets no probe. A driver that hangs without exiting is not restarted. Kubernetes recommends a liveness probe on the driver’s health socket.
When the driver restarts
A stepped story on gpu-node-1. Pick how long the new driver Pod takes to register; the kubelet’s 30 s wiping delay (defaultWipingDelay in pkg/kubelet/cm/dra/manager.go, release-1.35) decides which ending you get.
- 0 · HealthyA new GPU Pod is bound here; the kubelet prepares its claim through
dra.sock. - 1 · Old driver Pod stops
maxSurge: 0: the old Pod is terminated before the new one is created. Its sockets go; the kubelet deregistersgpu.amd.comand starts the wipe timer. - 2 · Gap: prepares failA Pod that starts now cannot be prepared and waits in
ContainerCreating. Pods ending now stayTerminating(no unprepare). - 3 · Running Pods carry on
…-p4x7kkeepscard5andrenderD132: they were added at container create. The driver is not in the data path. - 4 · Back in timeThe new driver reads
checkpoint.json, re-registers, and the kubelet cancels the wipe and retries the prepare. Nothing changed in the API.
- 0 · HealthyA new GPU Pod is bound here; the kubelet prepares its claim through
dra.sock. - 1 · Gap, longer than 30 sImage pull from a slow registry, a crash loop, or
amdgpunot loaded (init container loops). Prepares keep failing. - 2 · t = 30 s: slice wipeThe kubelet deletes all ResourceSlices for this node and driver (
DeleteCollection, field selector on node name and driver). - 3 · The scheduler stops sending GPU PodsNew GPU Pods go to other nodes or stay Pending. Running Pods are still untouched.
- 4 · Driver back: republishThe new driver registers and republishes its slice; the node’s GPUs are schedulable again.
- 5 · Pending prepares succeedThe kubelet’s retried prepare goes through; the waiting Pod starts.
0 s: old Pod stops
Images come from registry.local with IfNotPresent, so a plain restart is usually well under 30 s (in the example, the last driver Pod went from creation to running in about 1 s).
A kubelet restart is worse for visibility: at start-up kubelet 1.35 deletes all ResourceSlices of its node, for every driver, then waits for drivers to re-register and republish. The node’s GPUs vanish from the API for a few seconds.
Why no overlap
Kubelet 1.33+ supports seamless upgrades: two instances of a driver run side by side with socket names made unique by the Pod UID, and the kubelet may use either. A driver opts in with the helper’s RollingUpdate(podUID) option and a DaemonSet that allows surge. The AMD driver v1.0.1 does not pass that option, so it uses fixed socket names, and maxSurge: 0 keeps two Pods from fighting over them. The cost is the gap above, on one node at a time (maxUnavailable: 1).
Draining and maintenance
- Stop the DRA driver last. If it goes before the GPU Pods, they cannot be unprepared and hang in
Terminating. Kubernetes’ guidance is to check that no allocated or reserved claims remain on the node before stopping the driver. kubectl drainskips DaemonSet Pods (with--ignore-daemonsets), so the driver stays. Custom drain scripts that delete everything inkube-amd-gpuget this wrong.- Readiness gate. A node-level readiness check can verify that the kubelet has
gpu.amd.comregistered and that complete ResourceSlice pools list as many distinct PCI addresses as the GPU model expects (8 for MI355X), and taint the nodeNoScheduleuntil they do.
Metrics
| Metric | From | What it shows |
|---|---|---|
dra_operations_duration_seconds{operation_name="PrepareResources"} |
kubelet | time per prepare, from the kubelet’s side |
dra_operations_duration_seconds{operation_name="UnprepareResources"} |
kubelet | time per unprepare |
dra_grpc_operations_duration_seconds{method_name=~".*NodePrepareResources"} |
driver (helper library) | time inside the driver |
NVIDIA
NVIDIA’s DRA driver (gpu.nvidia.com) uses the same kubelet protocol and helper library. Differences: its CDI specs also mount host driver libraries and run hooks, it publishes MIG devices as mig.nvidia.com, and its ComputeDomain kubelet plugin prepares IMEX channels for NVLink across trays. See GPU operators.
Look for yourself
# on the GPU node (read-only)
ls -la /var/lib/kubelet/plugins_registry /var/lib/kubelet/plugins/gpu.amd.com /var/run/cdi
python3 -m json.tool /var/lib/kubelet/plugins/gpu.amd.com/checkpoint.json | head -40
journalctl -u kubelet --since -1h | grep -iE 'resourceclaim|NodePrepare|dra'
# from the management node
kubectl -n kube-amd-gpu get ds default-dra-driver -o wide
kubectl -n kube-amd-gpu logs <dra-driver-pod> -c plugin | head -20
kubectl get resourceslices --field-selector spec.nodeName=<node>
Further reading
- How DRA works — Kubernetes docs
- Good practices for DRA cluster admins: driver upgrades, liveness, draining, metrics — Kubernetes docs
- kubeletplugin helper package · DRA gRPC API (v1)
- AMD GPU DRA driver · troubleshooting