Learn · GPUs in Kubernetes

DRA on the node: registration, prepare, checkpoints and restarts

Updated 10 Nov 2026 · 8 min read

The scheduler only records an allocation in a ResourceClaim. On the node, the kubelet and the vendor’s kubelet plugin turn that record into device files in a container. This page follows that path on gpu-node-1, with the AMD DRA driver v1.0.1 and kubelet 1.35 (illustrative example output).

The idea

Think of the kubelet as a hotel front desk and the DRA driver as the GPU specialist on call. The specialist leaves a phone number in the desk’s directory (a socket in plugins_registry). When a guest (a Pod) arrives with a booking (an allocated claim), the desk calls the specialist: “prepare room gpu-5-132 for claim c1a10001”. The specialist sets it up and hands back a key card (the CDI names). Both keep a ledger of who is in which room, so either can be restarted without forgetting.

Piece Question it answers
Registration socket How does the kubelet find the driver at all?
ResourceSlice What GPUs exist on this node? (published by the driver)
NodePrepareResources Make this claim’s device usable; what should the container get?
Two checkpoints After a restart, which claims are already prepared?
30 s wiping delay How long does the kubelet trust a driver that went away?

Where the analogy breaks: once the container is running, the specialist is not needed. The device files stay even if the driver dies. The driver matters only when a container is created or a claim is released.

Devices published
8
one ResourceSlice for the node
Claims prepared now
2
gpu-6-133, gpu-5-132
Driver Pod start
~1 s
example kubelet log
Prepare timeout
45 s
kubelet default per call
Wiping delay
30 s
kubelet 1.35 defaultWipingDelay

Worked example: the kubelet prepares claim c1a10001

The same prefill Pod as on GPU allocation and CDI. Files and log lines are illustrative example output in the driver’s format; the two gRPC messages are rebuilt from the checkpoints (same fields as api.proto), since the wire messages are not logged.

From driver start to a prepared GPU

1 · 1 Nov 08:00: the driver finds 8 GPUs

I1101 08:00:00.100000 main.go:151] Feature gates: ContextualLogging=true
I1101 08:00:00.200000 discovery.go:136] Found full AMD GPU: gpu-3-130, compute type: spx, memory type: nps1
I1101 08:00:00.200100 discovery.go:136] Found full AMD GPU: gpu-4-131, compute type: spx, memory type: nps1
I1101 08:00:00.200200 discovery.go:136] Found full AMD GPU: gpu-5-132, compute type: spx, memory type: nps1
…
I1101 08:00:00.201000 discovery.go:177] Discovered 8 AMD GPU devices
I1101 08:00:00.300000 driver.go:106] Publishing ResourceSlice: …

The init container waited for amdgpu; then gpu-kubeletplugin read sysfs. Each GPU is named gpu-<card>-<render> and is a whole GPU (SPX compute, NPS1 memory).

2 · It opens two sockets

$ ls -la /var/lib/kubelet/plugins_registry /var/lib/kubelet/plugins/gpu.amd.com
srwxr-xr-x. 1 root root    0 Nov  1 08:00 gpu.amd.com-reg.sock   # kubelet watches this dir
srwxr-xr-x. 1 root root    0 Nov  1 08:00 dra.sock               # the gRPC service
-rw-------. 1 root root 1200 Nov  3 10:13 checkpoint.json        # driver's prepared claims

dra.sock is the service. gpu.amd.com-reg.sock in plugins_registry is the “phone number”: the kubelet’s plugin watcher (the same mechanism CSI uses) sees it appear. Socket times = last driver start.

3 · The kubelet calls GetInfo and registers the plugin

# kubelet → gpu.amd.com-reg.sock: GetInfo()  (PluginInfo fields; values reconstructed)
type: DRAPlugin
name: gpu.amd.com
endpoint: /var/lib/kubelet/plugins/gpu.amd.com/dra.sock
supported_versions: [v1.DRAPlugin, v1beta1.DRAPlugin]   # kubelet 1.35 picks v1

The reply names the driver, the endpoint and the gRPC services, as <package>.<service>. The kubelet picks one it supports. From now on any claim with driver gpu.amd.com on this node goes to dra.sock.

4 · The driver publishes the ResourceSlice

# ResourceSlice 00000-gpu.amd.com-gpu-node-1-x7q2m (1 of 8 devices)
- name: gpu-5-132
  attributes:
    productName: {string: Instinct_MI355X_OAM}
    partitionProfile: {string: spx_nps1}
    numaNode: {int: 1}
    resource.kubernetes.io/pciBusID: {string: "0000:8b:00.0"}
  capacity:
    memory: {value: 294912Mi}
    computeUnits: {value: "256"}

Written through the API server with the driver’s ServiceAccount amd-gpu-operator-dra-driver. The scheduler reads this; it picked gpu-5-132 (PCI 0000:8b:00.0, NUMA 1) for the claim.

5 · 3 Nov: the Pod is bound; the kubelet writes its ledger first

# /var/lib/kubelet/dra_manager_state (kind DRACheckpoint), this claim only
ClaimUID:  c1a10001-0000-4000-8000-000000000001
ClaimName: llm-prefill-5f7d8c9b6-p4x7k-gpu-z8k3q
Namespace: gpu-demo
PodUIDs:   {b0d10001-0000-4000-8000-000000000001: {}}
DriverState: {}        # filled in after the driver answers (step 6)

The kubelet checks the claim is allocated and reserved for this Pod, then records it before calling the driver, so a crash in between does not lose track of it.

6 · NodePrepareResources: request

# NodePrepareResourcesRequest → dra.sock
claims:
- namespace: gpu-demo
  uid: c1a10001-0000-4000-8000-000000000001
  name: llm-prefill-5f7d8c9b6-p4x7k-gpu-z8k3q

Only the claim’s identity travels; the driver fetches the claim and reads status.allocation for its own driver name. Calls are batched per Pod and, through the kubeletplugin helper, serialized: two prepares never run at once. Timeout 45 s.

7 · The driver writes the CDI file and its own ledger, then answers

/var/lib/kubelet/plugins/gpu.amd.com/checkpoint.json (trimmed)
"preparedClaims": {
  "c1a10001-0000-4000-8000-000000000001": [{
    "request_names": ["gpu"],
    "pool_name": "gpu-node-1",
    "device_name": "gpu-5-132",
    "cdi_device_ids": [
      "k8s.gpu.amd.com/gpu=common",
      "k8s.gpu.amd.com/gpu=c1a10001-0000-4000-8000-000000000001-gpu-5-132"
    ]
  }]
}
# NodePrepareResourcesResponse ← dra.sock
claims:
  c1a10001-0000-4000-8000-000000000001:            # keyed by claim UID
    devices:
    - request_names: [gpu]
      pool_name: gpu-node-1
      device_name: gpu-5-132
      cdi_device_ids:
      - k8s.gpu.amd.com/gpu=common
      - k8s.gpu.amd.com/gpu=c1a10001-0000-4000-8000-000000000001-gpu-5-132
    error: ""

The CDI file is k8s.gpu.amd.com-gpu_c1a10001-0000-4000-8000-000000000001.yaml (what is in it). Prepare is idempotent: asked again (say after a kubelet restart), the driver finds the UID in checkpoint.json and returns the same answer.

8 · The kubelet stores the CDI names and creates the container

DriverState:
  gpu.amd.com:
    Devices:
    - PoolName: gpu-node-1
      DeviceName: gpu-5-132
      RequestNames: [gpu]
      CDIDeviceIDs: [k8s.gpu.amd.com/gpu=common, k8s.gpu.amd.com/gpu=c1a10001…-gpu-5-132]

The kubelet passes the two names to containerd in CreateContainer for every container that lists the claim. The container started at 10:12:00 (node clock); the model took 4 min 0 s more to load.

9 · Health check: four records agree

RecordWhereClaims for gpu-node-1
API: claims with pool = nodekubectl get resourceclaims -A2
kubelet: dra_manager_stateClaimInfoStateList2
driver: checkpoint.jsonv1.preparedClaims2
runtime: /var/run/cdiper-claim files2

All four list the same 2 claims (gpu-6-133, gpu-5-132); the other 6 GPUs are free. A mismatch means a leak (a CDI file or checkpoint entry with no claim) or a lost prepare. Commands are under Look for yourself.

gpu-node-1, AMD DRA driver v1.0.1, kubelet 1.35. Driver start 1 Nov; Pod start 3 Nov 10:12 (node clock). Example output.

The whole conversation, including unprepare, as one sequence:

Registration, Pod start and Pod stop
PREPARE (POD START)REGISTER (DRIVER START)UNPREPARE (POD END)DRA driver (kubelet plugin)kubeletAPI servercontainerddiscover 8 GPUs in sysfs; open dra.sockcreateplugins_registry/gpu.amd.com-reg.sockGetInfo → name gpu.amd.com, endpoint,servicespublish ResourceSlice (8 devices)Pod bound here, claim allocatedcheckpoint claim in dra_manager_stateNodePrepareResources(claim uid, name,ns)get ResourceClaim (read allocation)write CDI spec + checkpoint.jsondevices + CDI IDsCreateContainer with CDI IDslast Pod gone: NodeUnprepareResourcesdelete CDI spec, update checkpoint.json

Prepare fails (driver down, timeout, error): the Pod stays in ContainerCreating with an event, and the kubelet retries. Nothing is rescheduled: the claim is already allocated to this node. The helper documentation also asks the driver to be the last line of defence and refuse a device already prepared for another claim.

Unprepare runs when the last Pod on the node that uses a claim has stopped. The driver deletes the CDI file and the checkpoint.json entry; the kubelet drops its record, and the resourceclaim controller frees the allocation in the API. A kubelet reconcile loop (every 60 s) retries unprepare for claims whose Pods are gone. Until unprepare succeeds the Pod is not reported terminated, so the GPU cannot be given to someone else.

Both checkpoint files carry a checksum, so a hand-edited or truncated file fails the check.

The driver Pod

The AMD GPU operator renders the DRA driver as DaemonSet kube-amd-gpu/default-dra-driver. The parts that matter (example, trimmed):

spec:
  updateStrategy:
    rollingUpdate: {maxSurge: 0, maxUnavailable: 1}   # old Pod stops before new starts
  template:
    spec:
      priorityClassName: system-node-critical
      nodeSelector: {example.com/gpu-vendor: AMD}
      tolerations: [{operator: Exists}]               # runs even on tainted nodes
      initContainers:
      - name: driver-init                              # wait for the amdgpu module
        command: [sh, -c, 'while [ ! -d /sys/class/kfd ] || [ ! -d /sys/module/amdgpu/drivers/ ]; do echo "amdgpu driver is not loaded "; sleep 2; done']
      containers:
      - name: plugin
        image: registry.local/rocm/k8s-gpu-dra-driver:v1.0.1
        command: [gpu-kubeletplugin]
        env:
        - {name: CDI_ROOT, value: /var/run/cdi}
        - {name: KUBELET_REGISTRAR_DIRECTORY_PATH, value: /var/lib/kubelet/plugins_registry}
        - {name: KUBELET_PLUGINS_DIRECTORY_PATH, value: /var/lib/kubelet/plugins}
        - name: NODE_NAME
          valueFrom: {fieldRef: {fieldPath: spec.nodeName}}
        securityContext: {privileged: true}
        volumeMounts:          # all hostPath
        - {name: plugins-registry, mountPath: /var/lib/kubelet/plugins_registry}
        - {name: plugins,          mountPath: /var/lib/kubelet/plugins}
        - {name: cdi,              mountPath: /var/run/cdi}
        - {name: dev,              mountPath: /dev}
        - {name: sys,              mountPath: /sys}

Read it as a list of facts:

  • It needs the host driver first. The init container loops until /sys/class/kfd and /sys/module/amdgpu/drivers/ exist. A node with a broken amdgpu install shows the driver Pod stuck in Init:0/1 and no ResourceSlice.
  • It runs everywhere it is selected. operator: Exists tolerates every taint, so tainting a GPU node (for example with a not-ready taint) does not stop the DRA driver. That is deliberate: Pods that are already running must still be unprepared when they end.
  • It is privileged and mounts /dev and /sys from the host, because it reads GPU topology from sysfs.
  • No liveness probe. The driver has a gRPC health server, but its --healthcheck-port defaults to -1 (off) and the DaemonSet sets no probe. A driver that hangs without exiting is not restarted. Kubernetes recommends a liveness probe on the driver’s health socket.

When the driver restarts

A stepped story on gpu-node-1. Pick how long the new driver Pod takes to register; the kubelet’s 30 s wiping delay (defaultWipingDelay in pkg/kubelet/cm/dra/manager.go, release-1.35) decides which ending you get.

New driver registers
Restart inside the window
GPU NODE GPU-NODE-1CONTROL PLANEDRA driver Podregistered, dra.sock openkubeletplugin gpu.amd.com knownRunning Pod …-p4x7khas card5 + renderD132New GPU Podbound here, claim allocatedResourceSlice8 devices of gpu-node-1Schedulerplaces GPU PodsNodePrepare Resourcesstart mepublishwipe after 30 sread free GPUs
  1. 0 · HealthyA new GPU Pod is bound here; the kubelet prepares its claim through dra.sock.
  2. 1 · Old driver Pod stopsmaxSurge: 0: the old Pod is terminated before the new one is created. Its sockets go; the kubelet deregisters gpu.amd.com and starts the wipe timer.
  3. 2 · Gap: prepares failA Pod that starts now cannot be prepared and waits in ContainerCreating. Pods ending now stay Terminating (no unprepare).
  4. 3 · Running Pods carry on…-p4x7k keeps card5 and renderD132: they were added at container create. The driver is not in the data path.
  5. 4 · Back in timeThe new driver reads checkpoint.json, re-registers, and the kubelet cancels the wipe and retries the prepare. Nothing changed in the API.
DRA driver outage on gpu-node-1

0 s: old Pod stops

ResourceSlice wiped?
GPUs hidden from the scheduler
New GPU Pods here wait in ContainerCreating
Running GPU Pods affected

Images come from registry.local with IfNotPresent, so a plain restart is usually well under 30 s (in the example, the last driver Pod went from creation to running in about 1 s).

A kubelet restart is worse for visibility: at start-up kubelet 1.35 deletes all ResourceSlices of its node, for every driver, then waits for drivers to re-register and republish. The node’s GPUs vanish from the API for a few seconds.

Why no overlap

Kubelet 1.33+ supports seamless upgrades: two instances of a driver run side by side with socket names made unique by the Pod UID, and the kubelet may use either. A driver opts in with the helper’s RollingUpdate(podUID) option and a DaemonSet that allows surge. The AMD driver v1.0.1 does not pass that option, so it uses fixed socket names, and maxSurge: 0 keeps two Pods from fighting over them. The cost is the gap above, on one node at a time (maxUnavailable: 1).

Draining and maintenance

  • Stop the DRA driver last. If it goes before the GPU Pods, they cannot be unprepared and hang in Terminating. Kubernetes’ guidance is to check that no allocated or reserved claims remain on the node before stopping the driver.
  • kubectl drain skips DaemonSet Pods (with --ignore-daemonsets), so the driver stays. Custom drain scripts that delete everything in kube-amd-gpu get this wrong.
  • Readiness gate. A node-level readiness check can verify that the kubelet has gpu.amd.com registered and that complete ResourceSlice pools list as many distinct PCI addresses as the GPU model expects (8 for MI355X), and taint the node NoSchedule until they do.

Metrics

Metric From What it shows
dra_operations_duration_seconds{operation_name="PrepareResources"} kubelet time per prepare, from the kubelet’s side
dra_operations_duration_seconds{operation_name="UnprepareResources"} kubelet time per unprepare
dra_grpc_operations_duration_seconds{method_name=~".*NodePrepareResources"} driver (helper library) time inside the driver

NVIDIA

NVIDIA’s DRA driver (gpu.nvidia.com) uses the same kubelet protocol and helper library. Differences: its CDI specs also mount host driver libraries and run hooks, it publishes MIG devices as mig.nvidia.com, and its ComputeDomain kubelet plugin prepares IMEX channels for NVLink across trays. See GPU operators.

Look for yourself

# on the GPU node (read-only)
ls -la /var/lib/kubelet/plugins_registry /var/lib/kubelet/plugins/gpu.amd.com /var/run/cdi
python3 -m json.tool /var/lib/kubelet/plugins/gpu.amd.com/checkpoint.json | head -40
journalctl -u kubelet --since -1h | grep -iE 'resourceclaim|NodePrepare|dra'
# from the management node
kubectl -n kube-amd-gpu get ds default-dra-driver -o wide
kubectl -n kube-amd-gpu logs <dra-driver-pod> -c plugin | head -20
kubectl get resourceslices --field-selector spec.nodeName=<node>

Further reading