Learn · Operating GPU nodes
GPU metrics and monitoring
GPU metrics come from a vendor exporter on every GPU node. Prometheus scrapes it like any other target, and GPUs become one more device type in the cluster dashboards. This page follows one counter from the chip to a query result, then covers what the exporter publishes and where it lies. Numbers are illustrative, for an example cluster of 4 nodes with 8× MI355X each (32 GPUs).
The idea
Think of a weather station. Instruments on the roof measure (the GPU’s firmware). A clerk reads the dials and writes the numbers on a board by the door (the exporter). Every 30 seconds someone copies the board into a notebook with the time (Prometheus). Later you ask questions of the notebook, like “how much rain fell this hour?” (PromQL).
Each part answers a different question:
- What is the value now? The board: one scrape of the exporter.
- What was it at 10:00? The notebook: stored samples.
- How fast is it changing? Arithmetic over samples:
rate(),increase(),changes(). - Is the board still being updated? Nobody checks this by default, and it is the question that matters most.
Where the analogy breaks, and why it matters: if the clerk falls asleep, the board still shows the last numbers, and the notebook keeps copying them every 30 seconds. Every page of the notebook looks normal. That is exactly what happens after some fatal GPU events: the exporter freezes at the moment of the fault, and every threshold check passes for hours.
- ReadThe exporter asks the driver (through the AMD SMI library) for every counter: temperatures, power, memory, ECC, energy.
- ScrapeEvery 30 s a Prometheus shard pulls
/metricsfrom the exporter and stores each value with a timestamp. - QueryYou ask
thanos-query, which fans out to both shards and merges the 2 replicas of each. Asking one shard directly gives only part of the 32 GPUs. - Frozen exporterThe GPU is dead, but the exporter still answers with the values it captured at the fault.
upstays 1. Only a check that asks did anything change? sees it.
| Stage | In the example cluster |
|---|---|
| Exporter | DaemonSet kube-amd-gpu/default-metrics-exporter, image device-metrics-exporter:v1.5.0 (example version), one Pod per GPU node (4), port 5000 |
| Scrape config | operator-made ServiceMonitor default-metrics-exporter, every 30 s, honorLabels: true |
| Prometheus | kube-prometheus-stack, 2 shards × 2 replicas (prometheus-prometheus-prometheus-*) |
| Query | thanos-query merges the shards; Grafana uses it |
Worked example: one counter, from the chip to a query
Follow gpu_energy_consumed for GPU 4 on gpu-node-1, the GPU an inference engine is using. Energy is a good teacher: it only goes up, so every step can be checked against the one before.
GPU 4's energy counter, from firmware to PromQL
1 · On the chip: amd-smi reads joules
$ amd-smi metric -g 4 -E -p # twice, 10 s apart## 10:00:00.000 UTC SOCKET_POWER: 520 W TOTAL_ENERGY_CONSUMPTION: 120000000.000 J1## 10:00:10.000 UTC SOCKET_POWER: 400 W TOTAL_ENERGY_CONSUMPTION: 120004200.000 J2- Energy so far, in joules. The GPU firmware counts it continuously.
- 10 s later it is 4,200 J higher: about 420 W on average.
The firmware keeps a running total of energy since power-on. Power readings (520 W, 400 W) are instant snapshots; the energy total integrates everything in between.
2 · On the exporter: the same counter in microjoules
$ kubectl get --raw /api/v1/namespaces/kube-amd-gpu/pods/default-metrics-exporter-x7k2p:5000/proxy/metrics# HELP gpu_energy_consumed Accumulated energy consumed by the GPU in uJgpu_energy_consumed{gpu_id="4",hostname="gpu-node-1",pod="llm-prefill-5d8f7c9b6-abcde",serial_number="SN-EXAMPLE-0005",…} 1.2000168e+141gpu_package_power{gpu_id="4",hostname="gpu-node-1",…} 430gpu_health{gpu_id="4",hostname="gpu-node-1",…} 12- Scraped at 10:00:04.000 UTC. 1.2000168e+14 µJ ÷ 1,000,000 = 120,001,680.0 J. amd-smi read 120,000,000.0 J at 10:00:00.000 and 120,004,200.0 J at 10:00:10.000; the scrape at 10:00:04.000 sits between them (checked).
gpu_healthrides along in the same scrape. It is derived from ECC counters, not from freshness.
The exporter publishes the counter as a gauge named gpu_energy_consumed, in µJ (its HELP line says so). Labels add the node, the GPU index, the board serial and, for a busy GPU, the Pod.
3 · In Prometheus: one sample every 30 s
| time (UTC) | energy (J) | Δ energy (J) | Δ time (s) | average W | power sample (W) |
|---|---|---|---|---|---|
| 09:55:05 | 119,888,790 | — | 370 | ||
| 09:55:17 | 119,888,790 | 0 | 12.0 | — | 370 |
| 09:55:47 | 119,900,190 | 11,400 | 30.0 | 380 | 400 |
| 09:56:17 | 119,912,490 | 12,300 | 30.0 | 410 | 640 |
| 09:56:47 | 119,925,990 | 13,500 | 30.0 | 450 | 380 |
| 09:57:17 | 119,937,090 | 11,100 | 30.0 | 370 | 350 |
| 09:57:47 | 119,947,890 | 10,800 | 30.0 | 360 | 360 |
| 09:58:17 | 119,959,140 | 11,250 | 30.0 | 375 | 390 |
| 09:58:47 | 119,971,140 | 12,000 | 30.0 | 400 | 560 |
| 09:59:17 | 119,982,990 | 11,850 | 30.0 | 395 | 380 |
| 09:59:47 | 119,994,540 | 11,550 | 30.0 | 385 | 390 |
The raw samples Thanos holds for the 5 minutes before 10:00:00, gpu_energy_consumed{hostname="gpu-node-1",gpu_id="4"}[5m]. Average watts per interval = Δ energy ÷ Δ time. The first two samples come from the two Prometheus replicas 12 s apart, before the value changed. Compare the columns: the power snapshots jump between 350 W and 640 W, while the energy-based averages stay between 360 W and 450 W.
4 · increase(): how much in the window
increase(gpu_energy_consumed{hostname="gpu-node-1",gpu_id="4"}[5m])Last sample − first sample = 105,750 J over 282.000 s. The samples do not reach the window edges (5.0 s missing at the start, 13.0 s at the end), so Prometheus stretches the change to the full 300 s: 105,750 J × 300 ÷ 282.000 = 112,500 J. Thanos answered 112,500,000,000 µJ, the same number.
5 · rate(): from a counter to watts
rate(gpu_energy_consumed{hostname="gpu-node-1",gpu_id="4"}[5m]) / 1e6 # µJ/s ÷ 10⁶ = Wrate() is increase() divided by the window: 112,500 J ÷ 300 s = 375.0 W. Thanos returned 375.0 W. Now compare avg_over_time(gpu_package_power[5m]), the average of the 11 power snapshots: 417.3 W (Thanos: 417.3 W). The snapshots happened to catch two spikes (640 W, 560 W), so they overstate. For power, rate the energy counter; do not average the gauge.
6 · Up the hierarchy: one number per node
sum by (hostname) (max by (hostname, gpu_id) (rate(gpu_energy_consumed[5m]))) / 1e6| series | value | |
|---|---|---|
gpu-node-1 | 2250 W | |
gpu-node-2 | 2311 W | |
gpu-node-3 | 2480 W | |
gpu-node-4 | 2206 W |
max by (hostname, gpu_id) first collapses any duplicate series to one per GPU; then sum by (hostname) adds the 8 GPUs. Bars are scaled to 8 × 1400 W: every node draws about a fifth of its GPU power budget.
7 · Is it fresh? changes()
min by (hostname) (changes(gpu_energy_consumed[15m]))| series | value | |
|---|---|---|
gpu-node-1 | 29 | |
gpu-node-2 | 30 | |
gpu-node-3 | 29 | |
gpu-node-4 | 30 |
In 15 minutes the slowest GPU on each node changed its energy value 29–30 times: once per scrape. With a frozen exporter this number is 0 for as long as the freeze lasts, while up and gpu_health read 1. The alert is changes(gpu_energy_consumed[15m]) == 0 (with up == 1). Temperature does not work for this: on an idle GPU it is whole degrees and may not change for 15 minutes.
Illustrative values with made-up numbers; every derived number is computed from them.
PromQL explorer: prepared queries, example answers
Pick a question. Each result is an illustrative answer from thanos-query on the example cluster (4 nodes, 32 GPUs).
count(count by (hostname, gpu_id) (gpu_health))| series | value | |
|---|---|---|
(all) | 32 |
One series per GPU: 4 nodes × 8 = 32, so every GPU reports. The expected number comes from inventory, not from Prometheus.
count by (hostname) (gpu_health)| series | value | |
|---|---|---|
gpu-node-1 | 8 | |
gpu-node-2 | 8 | |
gpu-node-3 | 8 | |
gpu-node-4 | 8 |
Eight on every node. Alert on `< 8`. After a fatal event this can stay 8: a frozen exporter keeps serving all eight.
min by (hostname) (gpu_health)| series | value | |
|---|---|---|
gpu-node-1 | 1 | |
gpu-node-2 | 1 | |
gpu-node-3 | 1 | |
gpu-node-4 | 1 |
1 everywhere. `gpu_health` is computed from ECC counters, so it cannot see a fatal ATHUB event. It stays 1 through such an event.
max by (hostname) (gpu_junction_temperature)| series | value | |
|---|---|---|
gpu-node-1 | 50 °C | |
gpu-node-2 | 48 °C | |
gpu-node-3 | 52 °C | |
gpu-node-4 | 49 °C |
Bars are scaled to the 100 °C slowdown point. 48–52 °C is far from it. A busy day might peak around 75 °C.
sum by (hostname) (max by (hostname, gpu_id) (rate(gpu_energy_consumed[5m]))) / 1e6| series | value | |
|---|---|---|
gpu-node-1 | 2,250 W | |
gpu-node-2 | 2,310.5 W | |
gpu-node-3 | 2,480.2 W | |
gpu-node-4 | 2,205.8 W |
True average power of all 8 GPUs, from the energy counter (µJ/s ÷ 10⁶ = W). Bars are scaled to 8 × 1400 W. 2.2–2.5 kW per node: mostly idle draw (8 × ~260 W) plus 1–3 busy engines.
count by (hostname) (max by (hostname, gpu_id) (gpu_used_vram) > 10000)| series | value | |
|---|---|---|
gpu-node-1 | 2 | |
gpu-node-2 | 2 | |
gpu-node-3 | 3 | |
gpu-node-4 | 1 |
GPUs holding more than 10,000 MiB: 8 of 32. VRAM is the reliable "someone owns it" signal; activity % drops to 0 between requests.
max by (hostname) (gpu_violation_ppt_residency_percentage)| series | value | |
|---|---|---|
gpu-node-1 | 9 % | |
gpu-node-2 | 0 % | |
gpu-node-3 | 7 % | |
gpu-node-4 | 0 % |
Share of time a GPU was held below full clock by the 1400 W cap. 9 % and 7 % on two nodes with busy engines: normal under load, not a fault. Thermal residency above 0 would be a cooling problem.
min by (hostname) (changes(gpu_energy_consumed[15m]))| series | value | |
|---|---|---|
gpu-node-1 | 29 | |
gpu-node-2 | 30 | |
gpu-node-3 | 29 | |
gpu-node-4 | 30 |
How often the slowest GPU on each node changed its energy value in 15 minutes: 29–30 times, once per 30-s scrape. A frozen exporter gives 0. The alert form is `changes(gpu_energy_consumed[15m]) == 0`, which returned nothing (good).
sum(increase(gpu_ecc_correct_total[7d])) and sum(increase(gpu_ecc_uncorrect_total[7d]))| series | value | |
|---|---|---|
correctable | 0 | |
uncorrectable | 0 |
No memory error of any kind on 32 GPUs in a week. The fleet is new; a background correctable rate will appear with age.
max(scrape_samples_scraped{job=~".*metrics-exporter.*"})| series | value | |
|---|---|---|
largest scrape | 2,512 |
Samples in one exporter scrape (one node). A direct scrape returned about 2,500. After a fatal GPU event, hundreds of series can vanish: a drop against this baseline is an early, vendor-neutral signal.
Two more that work:
# Join GPU metrics to node state (hostname vs node): hottest GPU per cordoned node
max by (hostname) (gpu_junction_temperature)
and on (hostname)
label_replace(kube_node_spec_unschedulable == 1, "hostname", "$1", "node", "(.*)")
# empty: nothing cordoned (with == 0: all 4 nodes)
# Nodes reporting fewer than 8 GPUs (a GPU fell off, or the exporter lost it)
count by (hostname) (gpu_health) < 8 # empty
The ECC families are gauges in this exporter version, but they behave as counters; increase() works on them as long as they do not reset (a reboot resets them). Two traps when writing your own queries:
- Duplicate series. Attribution and scrape labels can split one GPU into several series. In the example cluster
count(gpu_used_vram)is exactly 32, and onlygpu_process_cu_occupancyhas one series per process. Collapse withmax by (hostname, gpu_id)anyway before you count or average, and neversumpercentages. - Absent is not zero. A filter that returns nothing may mean “no problem” or “no data”. Check the unfiltered count first.
To check if the exporter lies: scrape it directly
From the management node, read-only, no port-forward:
kubectl -n kube-amd-gpu get pods -o wide | grep metrics-exporter # pick the Pod on your node
kubectl get --raw /api/v1/namespaces/kube-amd-gpu/pods/<exporter-pod>:5000/proxy/metrics | grep '^gpu_health'
One scrape is about 130 metric families and about 2,500 samples. Two scrapes 60 s apart should differ in gpu_energy_consumed; a frozen exporter returns byte-identical scrapes.
AMD metric families
Every family from one scrape, grouped:
| Group | Families | Notes |
|---|---|---|
| Health | gpu_health |
1 healthy, 0 unhealthy. Derived from ECC counters; see the warning below |
| Inventory | gpu_nodes_total |
GPUs on the node (8); labelled only by hostname |
| Utilization | gpu_gfx_activity, gpu_umc_activity, gpu_gfx_busy_instantaneous, gpu_vcn_busy_instantaneous, gpu_jpeg_busy_instantaneous |
% busy; umc is memory-controller activity |
| Memory | gpu_used_vram, gpu_free_vram, gpu_total_vram, *_visible_vram, *_gtt, gpu_vram_max_bandwidth |
VRAM in MB per the HELP text |
| Thermal | gpu_junction_temperature, gpu_memory_temperature |
°C. No edge sensor on MI35x |
| Power and energy | gpu_package_power, gpu_power_usage, gpu_energy_consumed |
W, W, µJ. Energy only increases while powered |
| Clocks | gpu_clock (clock_type, clock_index), gpu_min_clock, gpu_max_clock |
MHz |
| ECC | gpu_ecc_{correct,uncorrect,deferred}_{total,<block>} for 19 blocks |
60 families; only 7 blocks ever report on MI35x (AMD RAS and ECC) |
| Throttling | gpu_violation_*_{accumulated,percentage} for ppt, socket_thermal, vr_thermal, hbm_thermal, processor_hot, low_utilization, gfx_clock_below_host_limit_* |
% of time held below max clock, and why |
| XGMI | gpu_xgmi_link_rx, gpu_xgmi_link_tx (link_index), gpu_xgmi_nbr_* |
KB transferred per link |
| PCIe | pcie_replay_count, pcie_recovery_count, pcie_replay_rollover_count, pcie_nack_sent_count, pcie_nack_received_count, pcie_speed, pcie_max_speed, pcie_bandwidth, pcie_bidirectional_bandwidth |
host link health |
| Process | gpu_process_cu_occupancy (process_id) |
one series per GPU process |
Labels
An example series, GPU 4 on gpu-node-1:
gpu_used_vram{card_model="102-EXAMPLE-00",container="prefill",deployment_mode="baremetal",
gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",
hostname="gpu-node-1",job_id="",job_partition="",job_user="",
namespace="gpu-demo",pod="llm-prefill-5d8f7c9b6-abcde",
serial_number="SN-EXAMPLE-0005"} 256000
| Label | Meaning |
|---|---|
hostname |
the node. kube-state-metrics calls it node, so joins need label_replace |
gpu_id |
amd-smi index (0–7). Not the slot, not the DRA device name |
serial_number |
board serial; what goes on an RMA |
card_model |
board model number (for example 102-EXAMPLE-00) |
gpu_compute_partition_type, gpu_memory_partition_type, gpu_partition_id |
spx, nps1, 0: whole GPUs |
pod, namespace, container |
the workload holding the GPU, when there is one |
job_id, job_user, job_partition |
Slurm attribution; always empty here |
There is no GPU UUID and no DRA device name or claim label. The only key shared with Kubernetes objects is hostname (plus pod when it is filled).
Pod attribution under DRA: filled for busy GPUs
A common belief is that the pod, namespace and container labels stay empty under DRA because the exporter cannot see DRA allocations. With a current exporter that is not true.
- In the example cluster, 8 of the 32 GPUs hold more than 10 GB of VRAM. All 8 carry a
podlabel:count(count by (hostname,gpu_id)(gpu_used_vram{pod!=""}))returns 8. - The 24 idle GPUs had empty labels, which is correct: nothing holds them.
How it works: the exporter mounts the kubelet pod-resources socket (/var/lib/kubelet/pod-resources, set by metricsExporter.podResourceAPISocketPath in the DeviceConfig). Since exporter v1.4.1 it falls back to gpu.amd.com DRA claims when no device-plugin allocation exists (per the exporter’s issue reports). Empty labels on idle GPUs, or on an older exporter, explain the belief.
Limits that remain:
- Shared claims. If two Pods share one ResourceClaim, only one Pod gets the labels, and which one can change between scrapes (device-metrics-exporter #665). One claim per Pod avoids it.
- No claim or device label. You cannot join a series to a ResourceClaim directly. Go through the
podlabel, or throughhostname+gpu_id+ the BDF map in GPU process ↔ Pod. - Blank does not mean free. The ResourceClaim is the allocation ledger; the label is a hint. A metric built from Kubernetes objects (claims and their devices) gives an exact workload-to-GPU map.
NVIDIA: dcgm-exporter
On NVIDIA nodes, dcgm-exporter publishes DCGM fields. The default metric names changed in 4.9.0 (released 2026-11-06). Check which version you run before copying a query or dashboard:
| Meaning | up to 4.8.x | 4.9.0 and later |
|---|---|---|
| GPU utilization | DCGM_FI_DEV_GPU_UTIL (%) |
DCGM_FI_DEV_GPU_UTIL_RATIO |
| Memory used | DCGM_FI_DEV_FB_USED |
DCGM_FI_DEV_FB_USED_BYTES |
| GPU temperature | DCGM_FI_DEV_GPU_TEMP |
DCGM_FI_DEV_GPU_TEMP_CELSIUS |
| Power | DCGM_FI_DEV_POWER_USAGE |
DCGM_FI_DEV_BOARD_POWER_WATTS |
| Energy | DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION |
DCGM_FI_DEV_GPU_ENERGY_JOULES_TOTAL |
| Last Xid | DCGM_FI_DEV_XID_ERRORS |
DCGM_FI_DEV_XID_ERROR |
| PCIe replays | DCGM_FI_DEV_PCIE_REPLAY_COUNTER |
DCGM_FI_DEV_PCIE_REPLAY_TOTAL |
| Row-remap failure | DCGM_FI_DEV_ROW_REMAP_FAILURE |
DCGM_FI_DEV_ROW_REMAP_FAILED |
In both versions the default counters file leaves the ECC totals, retired pages and NVLink error counters commented out. A health alert on DBE (double-bit errors) never fires until you enable those fields in the counters CSV. The Xid metric holds the last Xid value, not a count. dcgm-exporter can also add DRA labels (claim name, namespace, pool, device), per the exporter issue above; not verified here.
Exporters by platform
| Platform | GPU metrics | Node health | Other exporters worth adding |
|---|---|---|---|
| AMD MI355X | device-metrics-exporter | node-problem-detector with AMD rules | IPMI, scale-out NIC |
| NVIDIA DGX / HGX | dcgm-exporter | NVSentinel | IPMI, scale-out NIC |
| AMD Helios | AMD exporter | — | scale-up and scale-out NIC, DPU NIC, switch (gNMI) |
| NVIDIA NVL72 | dcgm-exporter | — | NVLink, ConnectX-8, BlueField-3 DPU, NVLink switch (gNMI) |
Dashboards
- Both exporters come with vendor Grafana dashboards (AMD ships Compute Node, GPU, Job, Overview and System views). Ship them with the cluster’s deploy, as code, so a fresh cluster gets them instead of relying on hand imports.
- Make GPU nodes first-class in the cluster’s existing node and workload dashboards, not only in separate GPU views.
Alerts: a starter set
A new cluster often has no alert rule that references any gpu_ metric: the only GPU-related paging comes from the workload’s own alerts. A first set, each checked against the frozen-exporter failure above:
| Proposed alert | Trigger | Catches a fatal event with a frozen exporter? |
|---|---|---|
| GpuFatalError, GpuKernelCrash | NPD conditions | KernelCrash yes (kernel log); FatalError no (no CPER) |
| GpuUncorrectableError | increase(gpu_ecc_uncorrect_total[5m]) > 0 |
no: no counter moved |
| GpuNodeWedge | count by (hostname)(gpu_health == 0) >= 2 |
no: gpu_health stayed 1 |
| GpuNodeDeviceCountLow | count by (hostname)(gpu_health) < 8 |
no: the frozen exporter still served 8 |
| GpuDieTempCritical / GpuMemTempHigh | > 100 °C / > 95 °C | no |
| GpuExporterDown | up == 0 |
no: up stayed 1 |
| GpuTelemetryStale | no fresh samples | yes, if it uses the energy counter |
So the freshness rule is not a nice-to-have: it is the only metric-side rule that sees a wedge. Decide the routing too: one clear “GPU hardware fault → page X” target.
Misconceptions
- “
up == 1means the GPU data is good.” It means the HTTP scrape worked. - “Pod labels are always empty under DRA.” Not with a current exporter: busy GPUs are labelled.
- “
gpu_healthcovers all faults.” It covers what moves ECC counters. - “Any Prometheus endpoint gives the full picture.” On a sharded Prometheus, use Thanos.