Learn · Operating GPU nodes

AMD GPU tools: amd-smi, rocm-smi, sysfs, RVS, AGFHC

Updated 10 Nov 2026 · 9 min read

This page is the toolbox for AMD GPU nodes. Every command runs on the GPU node (reach it with ssh from the management node). The output is illustrative: made-up but realistic values for one example 8× MI355X node called gpu-node-1, trimmed to the lines that matter.

The idea

A GPU node is like a server with eight very expensive, very hot disks in it. You want to know the same things you would ask about disks: which one is this, what model is it, how is it doing right now, who is using it, is it damaged, are its cables up? amd-smi is one tool that answers all six, the way lsblk, smartctl, iostat, lsof and ethtool do between them for storage and network.

Question amd-smi subcommand The disk-world equivalent
Which card is this? list -e ls -l /dev/disk/by-path
What is it, and what are its limits? static the spec sheet, smartctl -i
How is it doing now? metric, monitor iostat, sensors
Who is using it? process lsof /dev/sdX
Is it damaged? bad-pages, ras SMART reallocated sectors
Are its links up? xgmi, topology ethtool, ip link

Where the analogy breaks: disks fail one at a time, and you can reset one without touching the others. The eight GPUs in an MI355X node are wired into one shared fabric, called a hive. Resetting one resets all eight. That is why everything below starts read-only.

Under every tool sits one fact: only the amdgpu kernel driver talks to the chip. Every tool is a reader of what the driver exposes. Step through the diagram to see which path each tool takes.

Every AMD tool reads through the driver
KERNELTOOLS YOU RUNamd-smithe CLI you typedevice-metrics-exporter: the same library, polled by Prometheus every 30 s.metrics exporterPod, every scraperocminfoROCm runtime viewjournalctl -kreads the host log onlyAMD SMI librarylibamd_smisysfs files/sys/class/drm/card5/device/dev/kfdcompute interfaceamdgpu driverthe only thing that talks to thechipkernel logjournaldMI355X GPU 4firmware: PSP, SMUwrites events
  1. amd-smiamd-smi metric calls the AMD SMI library, which reads sysfs files that the driver fills by asking the GPU’s firmware.
  2. The metrics exporterThe exporter Pod uses the same library, on every Prometheus scrape (every 30 s). Same data, same path: if amd-smi is wrong, the metrics are wrong too.
  3. rocminforocminfo asks the ROCm runtime through /dev/kfd, the interface programs use to run work on the GPU. A GPU that amd-smi lists but rocminfo misses is visible to the driver and unusable by programs.
  4. The kernel logThe driver also writes events (errors, resets, hangs) to the kernel log. journalctl -k reads only the host’s log, so it still works when the GPU does not answer.
  5. A wedged GPUWhen the GPU’s firmware stops answering, amd-smi blocks. Run it under timeout 30, and read journalctl -k first.

What is on the node

GPUs per node
8
one XGMI hive
HBM per GPU
288 GiB
294912 "MB" in amd-smi
Power cap
1400 W
idle draw ≈ 250 W
Hotspot slowdown
100 °C
shutdown at 110 °C
Bad-page budget
128
retired HBM pages
Item Example version How to check
amd-smi tool and library 26.0.0 amd-smi version
ROCm (user-space runtime) 7.0.0 amd-smi version
amdgpu kernel driver 6.14.0 cat /sys/module/amdgpu/version
GPU firmware package amdgpu-firmware-1.0.0-example dpkg -l | grep amdgpu
RVS, TransferBench, AGFHC 1.2.0, 1.60.0, 1.30.0 dpkg -l | grep -iE 'rvs|transferbench|agfhc'

Worked example: reading one GPU, field by field

Follow GPU 4 on gpu-node-1, which a prefill engine is using. Each step is one command, example output trimmed, with the lines to read highlighted and explained. Common flags on every subcommand: -g <index|BDF|UUID|all> picks GPUs, --json or --csv gives machine-readable output, -h lists the subcommand’s flags.

GPU 4 on gpu-node-1, one command at a time

1 · Which card is it? amd-smi list -e

gpu-node-1
$ amd-smi list -e -g 4GPU: 4    BDF: 0000:8b:00.01    UUID: 00000000-0000-0000-0000-000000000005    KFD_ID: 45678    NODE_ID: 6    PARTITION_ID: 0    RENDER: renderD1322    CARD: card53    HSA_ID: 6    HIP_ID: 44    HIP_UUID: GPU-00000000-0000-0000-0000-000000000005    OAM_ID: 65
  1. BDF (PCI bus:device.function). The kernel log, the DRA ResourceSlice (pciBusID) and RMA tickets use it.
  2. Render node /dev/dri/renderD132: the device file a container gets.
  3. DRM card card5. With the render number it makes the DRA device name gpu-5-132.
  4. HIP ID: the number programs see. Inside a container it renumbers from 0.
  5. OAM slot 6: the physical slot on the baseboard, the number DC Ops sees. GPU index 4 is not slot 4.

One GPU has five names, and ops work is mostly translating between them. The index order follows the BDF, not the slot: in this example GPU 0 sits in slot 2. Use -e whenever you must point at hardware. The full join to Pods is in GPU process ↔ Pod.

2 · What is it, and where are its limits? amd-smi static

gpu-node-1
$ amd-smi static -g 4 -a -B -l -r -p -u -vGPU: 4    ASIC:        MARKET_NAME: AMD Instinct MI355X        NUM_COMPUTE_UNITS: 256        TARGET_GRAPHICS_VERSION: gfx950    LIMIT:        PPT0:            SOCKET_POWER_LIMIT: 1400 W        SLOWDOWN_EDGE_TEMPERATURE: N/A        SLOWDOWN_HOTSPOT_TEMPERATURE: 100 °C1        SLOWDOWN_VRAM_TEMPERATURE: 105 °C        SHUTDOWN_HOTSPOT_TEMPERATURE: 110 °C        SHUTDOWN_VRAM_TEMPERATURE: 115 °C    BOARD:        MODEL_NUMBER: 102-EXAMPLE-00        PRODUCT_SERIAL: SN-EXAMPLE-00052        PRODUCT_NAME: AMD Instinct MI355X OAM    RAS:        BAD_PAGE_THRESHOLD: 1283        BAD_PAGE_THRESHOLD_EXCEEDED: False        POISON_SCHEMA: ENABLED    PARTITION:        ACCELERATOR_PARTITION: SPX4        MEMORY_PARTITION: NPS1    NUMA:        NODE: 15    VRAM:        TYPE: HBM3E        SIZE: 294912 MB6        MAX_BANDWIDTH: 8192 GB/s
  1. Clocks slow down at 100 °C hotspot (junction) and 105 °C HBM. The GPU shuts down at 110 °C and 115 °C. A health check can alarm a little below these, for example at 100 °C junction and 95 °C memory.
  2. Board serial: what AMD wants on an RMA. The exporter label serial_number carries the same value.
  3. Up to 128 HBM pages may be retired before the GPU is condemned. See AMD RAS and ECC.
  4. SPX / NPS1: one whole GPU with one memory pool, the usual mode for one large model per GPU. See partitioning.
  5. The closest CPU NUMA node. A Pod pinned to far CPUs pays an extra hop on every host copy.
  6. 294912 MB is really MiB: 288.0 GiB (computed from sysfs in step 7).

static prints facts that do not change while the node runs: sections -a ASIC, -b bus, -I firmware image, -d driver, -v VRAM, -B board, -r RAS, -l limits, -u NUMA, -p partition. The driver reports the thermal trip points itself, so nobody needs to guess thresholds.

3 · Which GPUs are busy? amd-smi monitor

gpu-node-1
$ amd-smi monitor -p -t -u -m -eGPU  XCP  POWER  PWR_CAP   GPU_T   MEM_T   GFX_CLK   GFX%   MEM%  SINGLE_ECC  DOUBLE_ECC  PCIE_REPLAY  0    0  250 W   1400 W   45 °C   35 °C   100 MHz    0 %    0 %           0           0            01  3    0  255 W   1400 W   47 °C   34 °C   100 MHz    0 %    0 %           0           0            0  4    0  400 W   1400 W   50 °C   38 °C  2400 MHz   12 %    5 %           0           0            02  5    0  320 W   1400 W   48 °C   37 °C  2400 MHz    2 %    0 %           0           0            0  7    0  260 W   1400 W   46 °C   35 °C   100 MHz    0 %    0 %           0           0            0
  1. Idle: about 250 W even doing nothing, clock near 100 MHz.
  2. Busy: GPU 4 runs at 2400 MHz and 400 W. Its GFX% is only 12 %: prefill works in bursts.

monitor is the top of GPUs: one line per GPU, printed once (add -w 5 to repeat every 5 s, -q for processes). The columns that matter: power against the 1400 W cap, the two temperatures, and the ECC and PCIe replay counts, which should be 0. Rows 1, 2 and 6 (idle) are cut here.

4 · How is it doing? amd-smi metric

gpu-node-1
$ amd-smi metric -g 4 -p -t -P -e -E -m -vGPU: 4    POWER:        SOCKET_POWER: 320 W    TEMPERATURE:        EDGE: N/A1        HOTSPOT: 50 °C        MEM: 38 °C        HBM_STACKS: [34 °C, 35 °C, 35 °C, 36 °C, 36 °C, 37 °C, 38 °C, 35 °C, N/A, N/A, N/A, N/A]    PCIE:        WIDTH: 16        SPEED: 32 GT/s        REPLAY_COUNT: 02        L0_TO_RECOVERY_COUNT: 0    ECC:        TOTAL_CORRECTABLE_COUNT: 0        TOTAL_UNCORRECTABLE_COUNT: 03        TOTAL_DEFERRED_COUNT: 0    ENERGY:        TOTAL_ENERGY_CONSUMPTION: 120050000.000 J4    MEM_USAGE:        TOTAL_VRAM: 294912 MB        USED_VRAM: 256000 MB5    THROTTLE:        PPT_VIOLATION_STATUS: NOT ACTIVE6        SOCKET_THERMAL_VIOLATION_STATUS: NOT ACTIVE        HBM_THERMAL_VIOLATION_STATUS: NOT ACTIVE
  1. No edge sensor on MI35x. Use HOTSPOT. A tool that alarms on edge temperature reads the "unsupported" value (65535) and fires on healthy hardware; an easy mistake in a first set of health-check rules.
  2. PCIe replays: the host link retransmitting packets. Rising means a riser, retimer or slot problem.
  3. ECC totals, all 0. Per block: -k.
  4. Energy only goes up while the chip is powered: the best "is this data fresh?" signal (step 5).
  5. 256,000 MiB in use: the prefill engine pre-allocates its weights and KV cache. VRAM, not GFX%, is the reliable "someone owns this GPU" signal.
  6. Throttle status: why clocks are below maximum. PPT is the power cap; SOCKET_THERMAL and HBM_THERMAL are heat. NOT ACTIVE is good.

With no flags, metric prints everything (two screens). Narrow it: -u usage, -p power, -t temperature, -P PCIe, -e ECC totals, -k ECC per block, -m memory, -v throttling, -E energy, -x XGMI errors.

5 · Power bounces; energy tells the truth

gpu-node-1
$ amd-smi metric -g 4 -E -p     # twice, 10 s apart## 10:00:00.000 UTC        SOCKET_POWER: 520 W1        TOTAL_ENERGY_CONSUMPTION: 120000000.000 J## 10:00:10.000 UTC        SOCKET_POWER: 400 W2        TOTAL_ENERGY_CONSUMPTION: 120004200.000 J3
  1. First power reading: 520 W.
  2. Ten seconds later: 400 W. Each is an instant snapshot of a bursty load.
  3. The energy counter grew. Divide by the time between the reads to get the true average power.

Average power over those 10 seconds: (120004200.000 − 120000000.000) J ÷ 10.000 s = 4200.0 J ÷ 10.00 s ≈ 420 W. Neither snapshot (520 W, 400 W) was the average. Prometheus does the same division with rate(); GPU metrics follows this exact counter into a query.

6 · Who is using it? amd-smi process

gpu-node-1
$ amd-smi process -g 4GPU: 4    PROCESS_INFO:        NAME: /usr/bin/python3.12        PID: 412341        VRAM_MEM: 2.0 MB    PROCESS_INFO:        NAME: /usr/bin/python3.12        PID: 415672        VRAM_MEM: 240.0 GB3        SDMA_USAGE: 123456789 us        CU_OCCUPANCY: 0
  1. The prefill driver process (inference_api.prefill_driver.main): 2 MB of GPU memory.
  2. The vLLM engine (VLLM::EngineCore) that holds the memory.
  3. 240.0 GB of the 256,000 MiB in use belong to this one process.

PIDs are host PIDs, not container PIDs. Two processes per GPU is normal for vLLM. Process names may need root. To get from these PIDs to the Pod, see GPU process ↔ Pod.

7 · The files under the tools: sysfs

gpu-node-1
$ cd /sys/class/drm/card5/device; cat …serial_number: SN-EXAMPLE-0005current_link_speed: 32.0 GT/s PCIecurrent_link_width: 16mem_info_vram_total: 3092376453121mem_info_vram_used: 2684354560002current_compute_partition: SPXhwmon/hwmon5/temp2_input: 50000      (temp2_label: junction)3hwmon/hwmon5/temp3_input: 38000      (temp3_label: mem)hwmon/hwmon5/power1_input: 3200000004ras/aca_umc: ue: 0 ce: 0 de: 0ras/event_state: Fatal Error: count:0, last_seqno:05
  1. 309,237,645,312 B ÷ 1,048,576 = 294,912 MiB (amd-smi prints "294912 MB") = 288.0 GiB
  2. 268,435,456,000 B ÷ 1,048,576 = 256,000 MiB, 86.8 % of the card
  3. Millidegrees: 50000 ÷ 1000 = 50 °C junction; 38000 ÷ 1000 = 38 °C memory.
  4. Microwatts: 320,000,000 µW ÷ 1,000,000 = 320 W.
  5. No fatal events since boot.

amd-smi is a view over these files. They are cheap to read and keep working in scripts when the tools are slow. Units are raw: bytes, millidegrees, microwatts.

8 · Is it damaged? Are the links up?

gpu-node-1
$ amd-smi bad-pages -g 4GPU: 4    RETIRED: No bad pages found.1    PENDING: No bad pages found.    UN_RES: No bad pages found.
  1. Retired HBM pages, taken out of use after an uncorrectable error. Zero is normal. The list lives in an EEPROM on the board and survives reboots.
gpu-node-1
$ amd-smi xgmi -lXGMI LINK STATUS:       bdf           GPU0  GPU1  GPU2  GPU3  GPU4  GPU5  GPU6  GPU7GPU0   0000:11:00.0  SELF  U     U     U     U     U     U     UGPU3   0000:6b:00.0  U     U     U     SELF  U     U     U     UGPU4   0000:8b:00.0  U     U     U     U     SELF  U     U     U1GPU7   0000:e5:00.0  U     U     U     U     U     U     U     SELFLegend:  U / D / X = Link is Up / Down / Disabled
  1. Seven U: GPU 4 has a direct, up link to each of the other seven. A D means down: the GPU still works alone, but peer traffic and the hive reset are at risk.

GPU 4 is healthy: no damage, all links up. Pending pages are marked bad but not yet removed; unreservable (UN_RES) ones could not be removed.

Illustrative output with made-up values. Derived numbers are computed from the output shown.

Other subcommands

amd-smi --help on 26.0.0 lists: version, list, static, firmware, bad-pages, metric, process, event, topology, set, reset, monitor, xgmi, partition, ras, node, fabric. Beyond the walkthrough:

Subcommand What it shows (example) Read it for
firmware per-GPU firmware list, e.g. PSP_SOSDRV 00.00.00.10, PM 00.01.02.03, PLDM_BUNDLE 01.00.00.00 version drift across the 8 GPUs and across nodes after a repair
topology -t / -o / -b XGMI between every pair, 1 hop, NUMA bandwidth 50000-50000 all 8 GPUs connect directly: one hive
xgmi (no flags) bit rate 32 Gb/s, max bandwidth 512 Gb/s per link, read/write counters a slow or erroring link; metric -x for errors
partition NPS1, SPX, memory caps NPS1,NPS2 partition mode; other profiles (DPX, QPX, CPX) split a GPU into 2, 4 or 8
ras --cper, ras --afid --cper-file <f> error records and AMD Field IDs (needs sudo) evidence for AMD support; --cper --folder writes files
node baseboard (UBB) temperatures, retimer regulators, node power threshold 12000 W, node power management disabled the board around the GPUs

Two firmware pieces matter most when things break. The PSP (Platform Security Processor) boots the chip and loads every other firmware. The SMU, shown as PM, manages power and clocks and executes resets. amd-smi firmware only reads; flashing goes through the BMC (Redfish) during provisioning or maintenance. Details of ras and the missing records for fatal ATHUB events are in AMD RAS and ECC.

rocm-smi and rocminfo

rocm-smi is the older tool, kept for scripts. With no flags it gives a compact table:

$ rocm-smi                     # example output, trimmed
Device  Node  IDs              Temp        Power     Partitions          SCLK     MCLK     Fan  Perf  PwrCap   VRAM%  GPU%
              (DID,     GUID)  (Junction)  (Socket)  (Mem, Compute, ID)
0       2     0x75a3,   12345  45.0°C      250.0W    NPS1, SPX, 0        100Mhz   2000Mhz  0%   auto  1400.0W  0%     0%
4       6     0x75a3,   45678  50.0°C      320.0W    NPS1, SPX, 0        2400Mhz  2000Mhz  0%   auto  1400.0W  87%    0%

rocminfo lists agents: each CPU NUMA node first (two on the example node), then each GPU. A GPU in amd-smi list but missing from rocminfo points at the runtime layer (KFD, /dev/kfd permissions, a wedged device), not at the PCI link.

sysfs: the files under the tools

The full set used in this track (example values for GPU 4):

File (under /sys/class/drm/card5/device/) Example value Meaning
current_compute_partition, current_memory_partition SPX, NPS1 partition mode
mem_info_vram_total, mem_info_vram_used 309237645312, 268435456000 (bytes) HBM size and use
gpu_busy_percent 0 engine busy, %
serial_number, product_name SN-EXAMPLE-0005, AMD Instinct MI355X OAM board identity
current_link_speed, current_link_width 32.0 GT/s PCIe, 16 PCIe link; fewer lanes or lower speed is a degraded link
ras/features feature mask: 0xfffff which RAS blocks are on
ras/aca_umc (also aca_gfx, aca_xgmi_wafl, …) ue: 0 ce: 0 de: 0 per-block error counts on this driver
ras/event_state Fatal Error: count:0 fatal and poison events since boot
ras/gpu_vram_bad_pages empty bad pages, pfn : size : R/P/F
xgmi_hive_info/xgmi_hive_id (number) GPUs with the same id share a hive
xgmi_physical_id 6 position in the hive
hwmon/hwmon*/temp2_input, temp3_input 50000, 38000 junction, memory, in millidegrees
hwmon/hwmon*/power1_input 320000000 socket power, µW
aer_dev_correctable, aer_dev_nonfatal, aer_dev_fatal all 0 PCIe Advanced Error Reporting counters

The kernel documentation names the per-block files <block>_err_count. On recent amdgpu drivers (6.14.0 in this example) they do not exist; the counts are in ras/aca_<block> instead. Scripts that hard-code the old names silently read nothing.

The kernel log

journalctl -k is the most reliable record of a GPU fault. After a couple of weeks of uptime, dmesg can return no amdgpu lines at all (the ring buffer has wrapped), while journalctl -k still has the boot messages. Use journald.

journalctl -k --since -1d | grep -iE 'amdgpu.*(ras|reset|hang|timeout|fault|poison|fatal)'

A healthy boot shows one RAS line per GPU:

amdgpu 0000:e5:00.0: RAS INFO: ras initialized successfully, hardware ability[fffff] ras_mask[fffff]
amdgpu 0000:e5:00.0: AMDGPU RAS Is Ready.

Patterns that mean trouble:

Pattern Meaning
RAS Intr triggered / ERREVENT_ATHUB_INTERRUPT a fatal hardware event; on a failing GPU it can repeat thousands of times
GPU reset begin the driver started a reset; printed once
ring … timeout a queue stopped making progress: a hang
GPU reset succeeded, trying to resume only means the reinit path ran; not proof of recovery
PSP resume failed / -62 the GPU’s boot processor did not come back; needs a cold power cycle
bad page / poison memory errors being contained
AER: PCIe errors on the link

Validation and diagnostics (disruptive, described only)

These tools load the GPUs to the limit or push traffic between them. They need the node cordoned and drained, and the workload owner must confirm the GPUs are free first. They were not run for this page.

Tool What it does Runtime Notes
RVS (ROCm Validation Suite, rvs) modules that stress compute, memory, PCIe and peer links, and check results against limits minutes per module public docs; config files choose the tests
AGFHC (AMD GPU Field Health Check, /opt/amd/agfhc/agfhc) AMD’s field diagnostic, run before an RMA and for burn-in; adds multi-GPU XGMI tests single_pass about 6 min; a burn-in recipe about 6 h distributed by AMD; no public docs found. Recipes include hbm_lvl5, pcie_lvl2, all_lvl5
TransferBench measures copy bandwidth between GPUs and host minutes finds a slow XGMI link or PCIe path

Two limits to remember:

  • A clean AGFHC pass does not prove a part is healthy. An intermittent fault that comes every 3 days on average is caught by a 2-hour test only about 3 % of the time.
  • A passing burn-in is not a performance test. AGFHC checks health, not throughput.

Misconceptions

  • “amd-smi reset -G -g 3 resets only GPU 3.” On MI35x it resets the hive, all 8 GPUs, and kills every process holding /dev/kfd.
  • “GPU index = slot.” No. Use amd-smi list -e and the OAM ID.
  • “No lines in dmesg means no errors.” The ring buffer wraps. Read journalctl -k.
  • “0 % GFX means the GPU is free.” Look at VRAM and amd-smi process, then the claim.

Further reading