Learn · Operating GPU nodes
From a GPU process to its Pod (and back)
GPU tools speak in GPU indexes, PIDs and PCI addresses. Kubernetes speaks in Pods, ResourceClaims and DRA device names. In an incident you need to cross between them fast:
- “GPU 4 is throwing errors: whose Pod is that?”
- “This Pod is slow: which card is it on?”
- “The vendor wants a serial number for device
gpu-5-132.”
Every recipe below is read-only. The outputs are illustrative example output for an 8× MI355X node called gpu-node-1: realistic, but made up.
The idea
Think of a hotel. The front desk (Kubernetes) knows guests by name and keeps a register of who has which room: that is the ResourceClaim. The housekeeping staff (the GPU tools) know rooms by number and see who is inside right now: that is amd-smi process. Every guest carries a key card with their name on it: that is the process’s cgroup path, which contains the Pod UID.
So the join is two lookups that meet in the middle:
- Room → guest: who is in the room (PIDs), then read their key card (cgroup → Pod).
- Guest → room: look up the register (Pod → claim → device), then translate the register’s room code into the number on the door (device → BDF → GPU index and slot).
Where the analogy breaks: a hotel never has two numbering systems for the same room. A GPU has five (index, BDF, card/render node, DRA device name, physical slot), and none of them is the slot number DC Ops sees on the board. Also, someone can be in a room without being in the register: a privileged Pod or a host process that opened the device directly. When the register and housekeeping disagree, believe the register for “is it allocated” and housekeeping for “is it in use”.
- GPU → PodRoom to guest: amd-smi gives the host PIDs, the PID’s cgroup path holds the Pod UID, and the UID names the Pod.
- Pod → GPUGuest to room: the Pod names its ResourceClaim, the claim names the DRA device, the ResourceSlice gives that device’s PCI address, and amd-smi maps the address to an index.
- GPU → physical cardFor DC Ops and the vendor: BDF → OAM slot and board serial. Never send just “GPU 4”: indexes can change after a hardware swap.
Worked example: GPU 4 to its Pod and back
GPU 4 on gpu-node-1 → Pod → claim → card
1 · GPU index → host PIDs
$ amd-smi process -g 4GPU: 4 PROCESS_INFO: NAME: /usr/bin/python3.12 PID: 412341 VRAM_MEM: 2.0 MB PROCESS_INFO: NAME: /usr/bin/python3.12 PID: 415672 VRAM_MEM: 240.0 GB SDMA_USAGE: 123456789 us CU_OCCUPANCY: 0- The prefill driver process.
- The vLLM engine: it holds 240.0 GB of GPU memory.
These are host PIDs: the numbers the GPU node’s kernel uses, not the ones ps shows inside the container.
2 · PID → its key card: the cgroup path
$ cat /proc/41567/cgroup0::/kubepods/burstable/poda1b2c3d4-0000-4000-8000-000000000001/0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef1- Split it: QoS class burstable, Pod UID a1b2c3d4-0000-4000-8000-000000000001, container ID 0123456789ab… (64 hex digits).
Kubernetes puts every container in a cgroup named after its Pod. The kubelet writes the Pod UID with underscores in some layouts; here it has dashes, exactly as in the API. The other PID, 41234, has the same cgroup line: both processes are in one container.
3 · Container → Pod
$ crictl ps --id 0123456789abc -o json | python3 -c '…print(namespace, pod, container)'gpu-demo llm-prefill-6b7c8d9f5-abcde prefill1- Namespace, Pod and container name, from the container's labels.
From the management node, the UID alone is enough: kubectl get pods -A -o jsonpath='{range .items[?(@.metadata.uid=="…")]}{.metadata.namespace}/{.metadata.name}{"\n"}{end}' prints the same Pod. (crictl warns about a missing /etc/crictl.yaml on the node; the default endpoint still works.)
4 · Pod → its ResourceClaim
$ kubectl -n gpu-demo get pod llm-prefill-6b7c8d9f5-abcde -o jsonpath='{.status.resourceClaimStatuses}'[{"name":"gpu","resourceClaimName":"llm-prefill-6b7c8d9f5-abcde-gpu-fghij"}]1- The claim generated from the Pod's claim template: Pod name +
-gpu-+ a random suffix.
Now switch direction: from the Kubernetes side, the Pod tells you which claim holds its GPU.
5 · Claim → device, and the loop closes
$ kubectl -n gpu-demo get resourceclaim llm-prefill-6b7c8d9f5-abcde-gpu-fghij -o jsonpath='{.status.allocation.devices.results}{"\n"}{.status.reservedFor}'[{"device":"gpu-5-132","driver":"gpu.amd.com","pool":"gpu-node-1","request":"gpu"}]1[{"name":"llm-prefill-6b7c8d9f5-abcde","resource":"pods","uid":"a1b2c3d4-0000-4000-8000-000000000001"}]2- The allocation is the tuple (driver, pool, device). The pool is the node name.
reservedForcarries Pod UID a1b2c3d4-0000-4000-8000-000000000001: the same UID as the cgroup path in step 2.
The register (claim) and the key card (cgroup) agree. That is the proof that this Pod owns this GPU.
6 · DRA device name → device files
$ ls -l /proc/41567/fd | grep -E 'dri|kfd' | awk '{print $NF}' | sort | uniq -c 1 /dev/dri/renderD1321 1 /dev/kfd2- The AMD DRA driver names devices
gpu-<card>-<render>:gpu-5-132=card5+renderD132. The process holds exactly that render node (checked). /dev/kfd: the ROCm compute interface every GPU program opens.
If a process holds render nodes of GPUs it was not allocated, something bypassed DRA (a privileged Pod or a host process). That is worth a security look.
7 · Device → PCI address → slot and serial
$ kubectl get resourceslice -o jsonpath='{range .items[?(@.spec.nodeName=="gpu-node-1")]}{range .spec.devices[*]}{.name}{" "}{.attributes.resource\.kubernetes\.io/pciBusID.string}{" numa="}{.attributes.numaNode.int}{"\n"}{end}{end}'gpu-3-130 0000:4d:00.0 numa=0gpu-4-131 0000:6b:00.0 numa=0gpu-5-132 0000:8b:00.0 numa=11gpu-6-133 0000:a9:00.0 numa=1...gpu-5-132is PCI device0000:8b:00.0, NUMA node 1.
$ amd-smi list -e -g 0000:8b:00.0GPU: 4 BDF: 0000:8b:00.01 UUID: 00000000-0000-0000-0000-000000000005 KFD_ID: 45678 NODE_ID: 6 PARTITION_ID: 0 RENDER: renderD132 CARD: card5 HSA_ID: 6 HIP_ID: 4 HIP_UUID: GPU-00000000-0000-0000-0000-000000000005 OAM_ID: 62- Same BDF: this is GPU index 4.
- Physical slot 6.
amd-smi static -B -g 4gives board serial SN-EXAMPLE-0005.
Each device in the slice also carries driverVersion, partitionProfile (spx_nps1), productName and pcieRoot attributes. There is no health attribute.
Illustrative example output. The splits and cross-checks are computed from the output shown.
- Pod
- …-abcde
- gpu-demo, container prefill
- Claim → device
- gpu-5-132
- card5, renderD132
- PCI BDF
- 0000:8b:00.0
- GPU index 4, NUMA 1
- Physical
- OAM slot 6
- serial SN-EXAMPLE-0005
For an RMA, give the vendor the node, BDF, OAM slot and board serial, never just “GPU 4”.
The identifiers and where they meet
| World | Identifier | Example on gpu-node-1 |
|---|---|---|
| amd-smi / exporter | GPU index (gpu_id) |
4 |
| PCI | BDF | 0000:8b:00.0 |
| DRM | card / render node | card5, /dev/dri/renderD132 |
| DRA | device name in the ResourceSlice | gpu-5-132 |
| Hardware | OAM slot, board serial | slot 6, serial from amd-smi static -B |
| Linux | host PID, cgroup | 41567, /kubepods/burstable/pod<uid>/<container-id> |
| Kubernetes | Pod, Pod UID, ResourceClaim | llm-prefill-…-abcde, a1b2c3d4-…, …-gpu-fghij |
The joins:
- DRA device name → card and render node. The AMD DRA driver names devices
gpu-<card>-<render>, sogpu-5-132iscard5andrenderD132. - ResourceSlice → BDF. Each device carries the attribute
resource.kubernetes.io/pciBusID. - BDF → everything else.
amd-smi list -eprints index, BDF, card, render and OAM slot for each GPU. - PID → Pod. The process’s cgroup path contains the Pod UID and container ID.
- Pod → GPU. The ResourceClaim’s
status.allocationnames the device, andstatus.reservedFornames the Pod UID.
The whole node on one screen
This script prints one line per GPU: index, BDF, slot, DRA device name, PIDs, and Pod. Run it as root on the GPU node. It only reads (amd-smi, /proc, crictl ps).
#!/bin/bash
# gpumap.sh — read-only: GPU index, BDF, OAM slot, DRA device, PIDs, Pod
amd-smi list -e --json | python3 -c '
import json, sys, re, subprocess
procs = json.loads(subprocess.run(["amd-smi", "process", "--json"], capture_output=True, text=True).stdout)
pids = {g["gpu"]: [p["process_info"]["pid"] for p in g.get("process_list", [])
if isinstance(p, dict) and isinstance(p.get("process_info"), dict)] for g in procs}
def pod(pid):
ids = re.findall(r"[0-9a-f]{64}", open("/proc/%d/cgroup" % pid).read())
if not ids: return "-"
out = subprocess.run(["crictl", "ps", "--id", ids[-1], "-o", "json"], capture_output=True, text=True).stdout
c = json.loads(out)["containers"]
return "%s/%s" % (c[0]["labels"]["io.kubernetes.pod.namespace"], c[0]["labels"]["io.kubernetes.pod.name"]) if c else "-"
fmt = "%-4s%-14s%-5s%-13s%-18s%s"
print(fmt % ("GPU", "BDF", "OAM", "DRA device", "PIDs", "Pod"))
for g in json.load(sys.stdin):
ps = pids.get(g["gpu"], [])
dev = "gpu-%s-%s" % (g["card"][4:], g["render"][7:])
print(fmt % (g["gpu"], g["bdf"], g["oam_id"], dev, ",".join(map(str, ps)) or "-", pod(ps[0]) if ps else "-"))
'
$ ./gpumap.shGPU BDF OAM DRA device PIDs Pod0 0000:11:00.0 2 gpu-1-128 - -1 0000:2f:00.0 3 gpu-2-129 - -2 0000:4d:00.0 0 gpu-3-130 - -3 0000:6b:00.0 1 gpu-4-131 - -4 0000:8b:00.0 6 gpu-5-132 41234,41567 gpu-demo/llm-prefill-6b7c8d9f5-abcde5 0000:a9:00.0 7 gpu-6-133 42890 gpu-demo/llm-prefill-small-7f8e9d6c5-klmno6 0000:c7:00.0 4 gpu-7-134 - -7 0000:e5:00.0 5 gpu-8-135 - -Three things to notice:
- The index order follows the BDF, not the slot: GPU 0 sits in OAM slot 2, GPU 2 in slot 0. DC Ops works in slots.
- The DRA names follow the DRM numbers, not the index: card numbers start at 1 here (
card1is GPU 0), and on hosts where each GPU registers several DRM nodes they skip numbers. Do not guess the name from the index. - Only two of eight GPUs are busy. That matches the two ResourceClaims allocated from this node’s pool (see below).
All GPUs in use on a node, from the claim side
kubectl get resourceclaims -A -o json | python3 -c '
import json, sys
for c in json.load(sys.stdin)["items"]:
for r in (c.get("status", {}).get("allocation") or {}).get("devices", {}).get("results", []):
if r["pool"] == "gpu-node-1":
print(c["metadata"]["namespace"], c["metadata"]["name"], r["device"])'
On gpu-node-1 this would print two claims: one Pod on gpu-5-132 and another on gpu-6-133. That matches the map script exactly. The claim is the allocation ledger. When the ledger, amd-smi process and the exporter disagree, believe the ledger for “is it allocated” and amd-smi process for “is it in use”.
Shortcut: the exporter label. Busy GPUs carry the Pod name on their metrics (see GPU metrics):
gpu_used_vram{gpu_id="4",hostname="gpu-node-1",namespace="gpu-demo",
pod="llm-prefill-6b7c8d9f5-abcde",container="prefill",...} 256000
Use it for dashboards. In an incident, confirm with the claim: a blank label does not prove the GPU is free.
NVIDIA variant
| Step | AMD | NVIDIA |
|---|---|---|
| GPU → PIDs | amd-smi process -g N |
nvidia-smi --query-compute-apps=pid,gpu_uuid,used_memory --format=csv |
| PID → Pod | /proc/<pid>/cgroup, crictl |
the same |
| Device files | /dev/dri/renderD*, /dev/kfd |
/dev/nvidia<N>, /dev/nvidiactl, /dev/nvidia-uvm |
| Device ↔ BDF | amd-smi list -e |
nvidia-smi --query-gpu=index,uuid,pci.bus_id --format=csv |
| Claim → device | ResourceClaim status.allocation |
the same (NVIDIA DRA driver gpu.nvidia.com; not verified here) |
Logs around a GPU Pod
| Want | Command |
|---|---|
| Prefill engine log | kubectl -n gpu-demo logs <pod> -c prefill (add --previous after a crash) |
| Why it restarted | kubectl -n <ns> describe pod <pod>: last state, exit code (139 = segfault, for example when the GPU dies) |
| The node’s view | journalctl -k --since -1h | grep -i amdgpu on the GPU node |
| DRA driver | kubectl -n kube-amd-gpu logs <default-dra-driver-pod-on-node> |
| Exporter | kubectl -n kube-amd-gpu logs <default-metrics-exporter-pod-on-node> |
Further reading
- Kubernetes DRA · AMD SMI CLI · nvidia-smi
- On this site: DRA on the node (how the claim becomes device files)