Learn · Operating GPU nodes

From a GPU process to its Pod (and back)

Updated 10 Nov 2026 · 5 min read

GPU tools speak in GPU indexes, PIDs and PCI addresses. Kubernetes speaks in Pods, ResourceClaims and DRA device names. In an incident you need to cross between them fast:

  • “GPU 4 is throwing errors: whose Pod is that?”
  • “This Pod is slow: which card is it on?”
  • “The vendor wants a serial number for device gpu-5-132.”

Every recipe below is read-only. The outputs are illustrative example output for an 8× MI355X node called gpu-node-1: realistic, but made up.

The idea

Think of a hotel. The front desk (Kubernetes) knows guests by name and keeps a register of who has which room: that is the ResourceClaim. The housekeeping staff (the GPU tools) know rooms by number and see who is inside right now: that is amd-smi process. Every guest carries a key card with their name on it: that is the process’s cgroup path, which contains the Pod UID.

So the join is two lookups that meet in the middle:

  • Room → guest: who is in the room (PIDs), then read their key card (cgroup → Pod).
  • Guest → room: look up the register (Pod → claim → device), then translate the register’s room code into the number on the door (device → BDF → GPU index and slot).

Where the analogy breaks: a hotel never has two numbering systems for the same room. A GPU has five (index, BDF, card/render node, DRA device name, physical slot), and none of them is the slot number DC Ops sees on the board. Also, someone can be in a room without being in the register: a privileged Pod or a host process that opened the device directly. When the register and housekeeping disagree, believe the register for “is it allocated” and housekeeping for “is it in use”.

The join loop, with GPU 4's example values
ON THE GPU NODEIN THE KUBERNETES APIGPU index 4amd-smiHost PIDs41234, 41567cgroup pathpod UID + container IDPCI BDF0000:8b:00.0OAM slot 6board serialPodllm-prefill-…-abcdeResourceClaimthe allocation ledgerDRA devicegpu-5-132 in theResourceSliceamd-smi process/proc/PID/cgroupPod UIDresourceClaimStatusesstatus.allocationpciBusIDamd-smi list -e
  1. GPU → PodRoom to guest: amd-smi gives the host PIDs, the PID’s cgroup path holds the Pod UID, and the UID names the Pod.
  2. Pod → GPUGuest to room: the Pod names its ResourceClaim, the claim names the DRA device, the ResourceSlice gives that device’s PCI address, and amd-smi maps the address to an index.
  3. GPU → physical cardFor DC Ops and the vendor: BDF → OAM slot and board serial. Never send just “GPU 4”: indexes can change after a hardware swap.

Worked example: GPU 4 to its Pod and back

GPU 4 on gpu-node-1 → Pod → claim → card

1 · GPU index → host PIDs

gpu-node-1
$ amd-smi process -g 4GPU: 4    PROCESS_INFO:        NAME: /usr/bin/python3.12        PID: 412341        VRAM_MEM: 2.0 MB    PROCESS_INFO:        NAME: /usr/bin/python3.12        PID: 415672        VRAM_MEM: 240.0 GB        SDMA_USAGE: 123456789 us        CU_OCCUPANCY: 0
  1. The prefill driver process.
  2. The vLLM engine: it holds 240.0 GB of GPU memory.

These are host PIDs: the numbers the GPU node’s kernel uses, not the ones ps shows inside the container.

2 · PID → its key card: the cgroup path

gpu-node-1
$ cat /proc/41567/cgroup0::/kubepods/burstable/poda1b2c3d4-0000-4000-8000-000000000001/0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef1
  1. Split it: QoS class burstable, Pod UID a1b2c3d4-0000-4000-8000-000000000001, container ID 0123456789ab… (64 hex digits).

Kubernetes puts every container in a cgroup named after its Pod. The kubelet writes the Pod UID with underscores in some layouts; here it has dashes, exactly as in the API. The other PID, 41234, has the same cgroup line: both processes are in one container.

3 · Container → Pod

gpu-node-1
$ crictl ps --id 0123456789abc -o json | python3 -c '…print(namespace, pod, container)'gpu-demo llm-prefill-6b7c8d9f5-abcde prefill1
  1. Namespace, Pod and container name, from the container's labels.

From the management node, the UID alone is enough: kubectl get pods -A -o jsonpath='{range .items[?(@.metadata.uid=="…")]}{.metadata.namespace}/{.metadata.name}{"\n"}{end}' prints the same Pod. (crictl warns about a missing /etc/crictl.yaml on the node; the default endpoint still works.)

4 · Pod → its ResourceClaim

mgmt node
$ kubectl -n gpu-demo get pod llm-prefill-6b7c8d9f5-abcde -o jsonpath='{.status.resourceClaimStatuses}'[{"name":"gpu","resourceClaimName":"llm-prefill-6b7c8d9f5-abcde-gpu-fghij"}]1
  1. The claim generated from the Pod's claim template: Pod name + -gpu- + a random suffix.

Now switch direction: from the Kubernetes side, the Pod tells you which claim holds its GPU.

5 · Claim → device, and the loop closes

mgmt node
$ kubectl -n gpu-demo get resourceclaim llm-prefill-6b7c8d9f5-abcde-gpu-fghij -o jsonpath='{.status.allocation.devices.results}{"\n"}{.status.reservedFor}'[{"device":"gpu-5-132","driver":"gpu.amd.com","pool":"gpu-node-1","request":"gpu"}]1[{"name":"llm-prefill-6b7c8d9f5-abcde","resource":"pods","uid":"a1b2c3d4-0000-4000-8000-000000000001"}]2
  1. The allocation is the tuple (driver, pool, device). The pool is the node name.
  2. reservedFor carries Pod UID a1b2c3d4-0000-4000-8000-000000000001: the same UID as the cgroup path in step 2.

The register (claim) and the key card (cgroup) agree. That is the proof that this Pod owns this GPU.

6 · DRA device name → device files

gpu-node-1
$ ls -l /proc/41567/fd | grep -E 'dri|kfd' | awk '{print $NF}' | sort | uniq -c      1 /dev/dri/renderD1321      1 /dev/kfd2
  1. The AMD DRA driver names devices gpu-<card>-<render>: gpu-5-132 = card5 + renderD132. The process holds exactly that render node (checked).
  2. /dev/kfd: the ROCm compute interface every GPU program opens.

If a process holds render nodes of GPUs it was not allocated, something bypassed DRA (a privileged Pod or a host process). That is worth a security look.

7 · Device → PCI address → slot and serial

mgmt node
$ kubectl get resourceslice -o jsonpath='{range .items[?(@.spec.nodeName=="gpu-node-1")]}{range .spec.devices[*]}{.name}{" "}{.attributes.resource\.kubernetes\.io/pciBusID.string}{" numa="}{.attributes.numaNode.int}{"\n"}{end}{end}'gpu-3-130 0000:4d:00.0 numa=0gpu-4-131 0000:6b:00.0 numa=0gpu-5-132 0000:8b:00.0 numa=11gpu-6-133 0000:a9:00.0 numa=1...
  1. gpu-5-132 is PCI device 0000:8b:00.0, NUMA node 1.
gpu-node-1
$ amd-smi list -e -g 0000:8b:00.0GPU: 4    BDF: 0000:8b:00.01    UUID: 00000000-0000-0000-0000-000000000005    KFD_ID: 45678    NODE_ID: 6    PARTITION_ID: 0    RENDER: renderD132    CARD: card5    HSA_ID: 6    HIP_ID: 4    HIP_UUID: GPU-00000000-0000-0000-0000-000000000005    OAM_ID: 62
  1. Same BDF: this is GPU index 4.
  2. Physical slot 6. amd-smi static -B -g 4 gives board serial SN-EXAMPLE-0005.

Each device in the slice also carries driverVersion, partitionProfile (spx_nps1), productName and pcieRoot attributes. There is no health attribute.

Illustrative example output. The splits and cross-checks are computed from the output shown.

Pod
…-abcde
gpu-demo, container prefill
Claim → device
gpu-5-132
card5, renderD132
PCI BDF
0000:8b:00.0
GPU index 4, NUMA 1
Physical
OAM slot 6
serial SN-EXAMPLE-0005

For an RMA, give the vendor the node, BDF, OAM slot and board serial, never just “GPU 4”.

The identifiers and where they meet

World Identifier Example on gpu-node-1
amd-smi / exporter GPU index (gpu_id) 4
PCI BDF 0000:8b:00.0
DRM card / render node card5, /dev/dri/renderD132
DRA device name in the ResourceSlice gpu-5-132
Hardware OAM slot, board serial slot 6, serial from amd-smi static -B
Linux host PID, cgroup 41567, /kubepods/burstable/pod<uid>/<container-id>
Kubernetes Pod, Pod UID, ResourceClaim llm-prefill-…-abcde, a1b2c3d4-…, …-gpu-fghij

The joins:

  • DRA device name → card and render node. The AMD DRA driver names devices gpu-<card>-<render>, so gpu-5-132 is card5 and renderD132.
  • ResourceSlice → BDF. Each device carries the attribute resource.kubernetes.io/pciBusID.
  • BDF → everything else. amd-smi list -e prints index, BDF, card, render and OAM slot for each GPU.
  • PID → Pod. The process’s cgroup path contains the Pod UID and container ID.
  • Pod → GPU. The ResourceClaim’s status.allocation names the device, and status.reservedFor names the Pod UID.

The whole node on one screen

This script prints one line per GPU: index, BDF, slot, DRA device name, PIDs, and Pod. Run it as root on the GPU node. It only reads (amd-smi, /proc, crictl ps).

#!/bin/bash
# gpumap.sh — read-only: GPU index, BDF, OAM slot, DRA device, PIDs, Pod
amd-smi list -e --json | python3 -c '
import json, sys, re, subprocess
procs = json.loads(subprocess.run(["amd-smi", "process", "--json"], capture_output=True, text=True).stdout)
pids = {g["gpu"]: [p["process_info"]["pid"] for p in g.get("process_list", [])
                   if isinstance(p, dict) and isinstance(p.get("process_info"), dict)] for g in procs}
def pod(pid):
    ids = re.findall(r"[0-9a-f]{64}", open("/proc/%d/cgroup" % pid).read())
    if not ids: return "-"
    out = subprocess.run(["crictl", "ps", "--id", ids[-1], "-o", "json"], capture_output=True, text=True).stdout
    c = json.loads(out)["containers"]
    return "%s/%s" % (c[0]["labels"]["io.kubernetes.pod.namespace"], c[0]["labels"]["io.kubernetes.pod.name"]) if c else "-"
fmt = "%-4s%-14s%-5s%-13s%-18s%s"
print(fmt % ("GPU", "BDF", "OAM", "DRA device", "PIDs", "Pod"))
for g in json.load(sys.stdin):
    ps = pids.get(g["gpu"], [])
    dev = "gpu-%s-%s" % (g["card"][4:], g["render"][7:])
    print(fmt % (g["gpu"], g["bdf"], g["oam_id"], dev, ",".join(map(str, ps)) or "-", pod(ps[0]) if ps else "-"))
'
gpu-node-1
$ ./gpumap.shGPU BDF           OAM  DRA device   PIDs              Pod0   0000:11:00.0  2    gpu-1-128    -                 -1   0000:2f:00.0  3    gpu-2-129    -                 -2   0000:4d:00.0  0    gpu-3-130    -                 -3   0000:6b:00.0  1    gpu-4-131    -                 -4   0000:8b:00.0  6    gpu-5-132    41234,41567       gpu-demo/llm-prefill-6b7c8d9f5-abcde5   0000:a9:00.0  7    gpu-6-133    42890             gpu-demo/llm-prefill-small-7f8e9d6c5-klmno6   0000:c7:00.0  4    gpu-7-134    -                 -7   0000:e5:00.0  5    gpu-8-135    -                 -

Three things to notice:

  • The index order follows the BDF, not the slot: GPU 0 sits in OAM slot 2, GPU 2 in slot 0. DC Ops works in slots.
  • The DRA names follow the DRM numbers, not the index: card numbers start at 1 here (card1 is GPU 0), and on hosts where each GPU registers several DRM nodes they skip numbers. Do not guess the name from the index.
  • Only two of eight GPUs are busy. That matches the two ResourceClaims allocated from this node’s pool (see below).

All GPUs in use on a node, from the claim side

kubectl get resourceclaims -A -o json | python3 -c '
import json, sys
for c in json.load(sys.stdin)["items"]:
    for r in (c.get("status", {}).get("allocation") or {}).get("devices", {}).get("results", []):
        if r["pool"] == "gpu-node-1":
            print(c["metadata"]["namespace"], c["metadata"]["name"], r["device"])'

On gpu-node-1 this would print two claims: one Pod on gpu-5-132 and another on gpu-6-133. That matches the map script exactly. The claim is the allocation ledger. When the ledger, amd-smi process and the exporter disagree, believe the ledger for “is it allocated” and amd-smi process for “is it in use”.

Shortcut: the exporter label. Busy GPUs carry the Pod name on their metrics (see GPU metrics):

gpu_used_vram{gpu_id="4",hostname="gpu-node-1",namespace="gpu-demo",
  pod="llm-prefill-6b7c8d9f5-abcde",container="prefill",...} 256000

Use it for dashboards. In an incident, confirm with the claim: a blank label does not prove the GPU is free.

NVIDIA variant

Step AMD NVIDIA
GPU → PIDs amd-smi process -g N nvidia-smi --query-compute-apps=pid,gpu_uuid,used_memory --format=csv
PID → Pod /proc/<pid>/cgroup, crictl the same
Device files /dev/dri/renderD*, /dev/kfd /dev/nvidia<N>, /dev/nvidiactl, /dev/nvidia-uvm
Device ↔ BDF amd-smi list -e nvidia-smi --query-gpu=index,uuid,pci.bus_id --format=csv
Claim → device ResourceClaim status.allocation the same (NVIDIA DRA driver gpu.nvidia.com; not verified here)

Logs around a GPU Pod

Want Command
Prefill engine log kubectl -n gpu-demo logs <pod> -c prefill (add --previous after a crash)
Why it restarted kubectl -n <ns> describe pod <pod>: last state, exit code (139 = segfault, for example when the GPU dies)
The node’s view journalctl -k --since -1h | grep -i amdgpu on the GPU node
DRA driver kubectl -n kube-amd-gpu logs <default-dra-driver-pod-on-node>
Exporter kubectl -n kube-amd-gpu logs <default-metrics-exporter-pod-on-node>

Further reading