Learn · Operating GPU nodes
AMD GPU tools: amd-smi, rocm-smi, sysfs, RVS, AGFHC
This page is the toolbox for AMD GPU nodes. Every command runs on the GPU node (reach it with ssh from the management node). The output is illustrative: made-up but realistic values for one example 8× MI355X node called gpu-node-1, trimmed to the lines that matter.
The idea
A GPU node is like a server with eight very expensive, very hot disks in it. You want to know the same things you would ask about disks: which one is this, what model is it, how is it doing right now, who is using it, is it damaged, are its cables up? amd-smi is one tool that answers all six, the way lsblk, smartctl, iostat, lsof and ethtool do between them for storage and network.
| Question | amd-smi subcommand | The disk-world equivalent |
|---|---|---|
| Which card is this? | list -e |
ls -l /dev/disk/by-path |
| What is it, and what are its limits? | static |
the spec sheet, smartctl -i |
| How is it doing now? | metric, monitor |
iostat, sensors |
| Who is using it? | process |
lsof /dev/sdX |
| Is it damaged? | bad-pages, ras |
SMART reallocated sectors |
| Are its links up? | xgmi, topology |
ethtool, ip link |
Where the analogy breaks: disks fail one at a time, and you can reset one without touching the others. The eight GPUs in an MI355X node are wired into one shared fabric, called a hive. Resetting one resets all eight. That is why everything below starts read-only.
Under every tool sits one fact: only the amdgpu kernel driver talks to the chip. Every tool is a reader of what the driver exposes. Step through the diagram to see which path each tool takes.
- amd-smi
amd-smi metriccalls the AMD SMI library, which reads sysfs files that the driver fills by asking the GPU’s firmware. - The metrics exporterThe exporter Pod uses the same library, on every Prometheus scrape (every 30 s). Same data, same path: if amd-smi is wrong, the metrics are wrong too.
- rocminforocminfo asks the ROCm runtime through
/dev/kfd, the interface programs use to run work on the GPU. A GPU that amd-smi lists but rocminfo misses is visible to the driver and unusable by programs. - The kernel logThe driver also writes events (errors, resets, hangs) to the kernel log.
journalctl -kreads only the host’s log, so it still works when the GPU does not answer. - A wedged GPUWhen the GPU’s firmware stops answering, amd-smi blocks. Run it under
timeout 30, and readjournalctl -kfirst.
What is on the node
- GPUs per node
- 8
- one XGMI hive
- HBM per GPU
- 288 GiB
- 294912 "MB" in amd-smi
- Power cap
- 1400 W
- idle draw ≈ 250 W
- Hotspot slowdown
- 100 °C
- shutdown at 110 °C
- Bad-page budget
- 128
- retired HBM pages
| Item | Example version | How to check |
|---|---|---|
amd-smi tool and library |
26.0.0 | amd-smi version |
| ROCm (user-space runtime) | 7.0.0 | amd-smi version |
amdgpu kernel driver |
6.14.0 | cat /sys/module/amdgpu/version |
| GPU firmware package | amdgpu-firmware-1.0.0-example |
dpkg -l | grep amdgpu |
| RVS, TransferBench, AGFHC | 1.2.0, 1.60.0, 1.30.0 | dpkg -l | grep -iE 'rvs|transferbench|agfhc' |
Worked example: reading one GPU, field by field
Follow GPU 4 on gpu-node-1, which a prefill engine is using. Each step is one command, example output trimmed, with the lines to read highlighted and explained. Common flags on every subcommand: -g <index|BDF|UUID|all> picks GPUs, --json or --csv gives machine-readable output, -h lists the subcommand’s flags.
GPU 4 on gpu-node-1, one command at a time
1 · Which card is it? amd-smi list -e
$ amd-smi list -e -g 4GPU: 4 BDF: 0000:8b:00.01 UUID: 00000000-0000-0000-0000-000000000005 KFD_ID: 45678 NODE_ID: 6 PARTITION_ID: 0 RENDER: renderD1322 CARD: card53 HSA_ID: 6 HIP_ID: 44 HIP_UUID: GPU-00000000-0000-0000-0000-000000000005 OAM_ID: 65- BDF (PCI bus:device.function). The kernel log, the DRA ResourceSlice (
pciBusID) and RMA tickets use it. - Render node
/dev/dri/renderD132: the device file a container gets. - DRM card
card5. With the render number it makes the DRA device namegpu-5-132. - HIP ID: the number programs see. Inside a container it renumbers from 0.
- OAM slot 6: the physical slot on the baseboard, the number DC Ops sees. GPU index 4 is not slot 4.
One GPU has five names, and ops work is mostly translating between them. The index order follows the BDF, not the slot: in this example GPU 0 sits in slot 2. Use -e whenever you must point at hardware. The full join to Pods is in GPU process ↔ Pod.
2 · What is it, and where are its limits? amd-smi static
$ amd-smi static -g 4 -a -B -l -r -p -u -vGPU: 4 ASIC: MARKET_NAME: AMD Instinct MI355X NUM_COMPUTE_UNITS: 256 TARGET_GRAPHICS_VERSION: gfx950 LIMIT: PPT0: SOCKET_POWER_LIMIT: 1400 W SLOWDOWN_EDGE_TEMPERATURE: N/A SLOWDOWN_HOTSPOT_TEMPERATURE: 100 °C1 SLOWDOWN_VRAM_TEMPERATURE: 105 °C SHUTDOWN_HOTSPOT_TEMPERATURE: 110 °C SHUTDOWN_VRAM_TEMPERATURE: 115 °C BOARD: MODEL_NUMBER: 102-EXAMPLE-00 PRODUCT_SERIAL: SN-EXAMPLE-00052 PRODUCT_NAME: AMD Instinct MI355X OAM RAS: BAD_PAGE_THRESHOLD: 1283 BAD_PAGE_THRESHOLD_EXCEEDED: False POISON_SCHEMA: ENABLED PARTITION: ACCELERATOR_PARTITION: SPX4 MEMORY_PARTITION: NPS1 NUMA: NODE: 15 VRAM: TYPE: HBM3E SIZE: 294912 MB6 MAX_BANDWIDTH: 8192 GB/s- Clocks slow down at 100 °C hotspot (junction) and 105 °C HBM. The GPU shuts down at 110 °C and 115 °C. A health check can alarm a little below these, for example at 100 °C junction and 95 °C memory.
- Board serial: what AMD wants on an RMA. The exporter label
serial_numbercarries the same value. - Up to 128 HBM pages may be retired before the GPU is condemned. See AMD RAS and ECC.
- SPX / NPS1: one whole GPU with one memory pool, the usual mode for one large model per GPU. See partitioning.
- The closest CPU NUMA node. A Pod pinned to far CPUs pays an extra hop on every host copy.
- 294912 MB is really MiB: 288.0 GiB (computed from sysfs in step 7).
static prints facts that do not change while the node runs: sections -a ASIC, -b bus, -I firmware image, -d driver, -v VRAM, -B board, -r RAS, -l limits, -u NUMA, -p partition. The driver reports the thermal trip points itself, so nobody needs to guess thresholds.
3 · Which GPUs are busy? amd-smi monitor
$ amd-smi monitor -p -t -u -m -eGPU XCP POWER PWR_CAP GPU_T MEM_T GFX_CLK GFX% MEM% SINGLE_ECC DOUBLE_ECC PCIE_REPLAY 0 0 250 W 1400 W 45 °C 35 °C 100 MHz 0 % 0 % 0 0 01 3 0 255 W 1400 W 47 °C 34 °C 100 MHz 0 % 0 % 0 0 0 4 0 400 W 1400 W 50 °C 38 °C 2400 MHz 12 % 5 % 0 0 02 5 0 320 W 1400 W 48 °C 37 °C 2400 MHz 2 % 0 % 0 0 0 7 0 260 W 1400 W 46 °C 35 °C 100 MHz 0 % 0 % 0 0 0- Idle: about 250 W even doing nothing, clock near 100 MHz.
- Busy: GPU 4 runs at 2400 MHz and 400 W. Its
GFX%is only 12 %: prefill works in bursts.
monitor is the top of GPUs: one line per GPU, printed once (add -w 5 to repeat every 5 s, -q for processes). The columns that matter: power against the 1400 W cap, the two temperatures, and the ECC and PCIe replay counts, which should be 0. Rows 1, 2 and 6 (idle) are cut here.
4 · How is it doing? amd-smi metric
$ amd-smi metric -g 4 -p -t -P -e -E -m -vGPU: 4 POWER: SOCKET_POWER: 320 W TEMPERATURE: EDGE: N/A1 HOTSPOT: 50 °C MEM: 38 °C HBM_STACKS: [34 °C, 35 °C, 35 °C, 36 °C, 36 °C, 37 °C, 38 °C, 35 °C, N/A, N/A, N/A, N/A] PCIE: WIDTH: 16 SPEED: 32 GT/s REPLAY_COUNT: 02 L0_TO_RECOVERY_COUNT: 0 ECC: TOTAL_CORRECTABLE_COUNT: 0 TOTAL_UNCORRECTABLE_COUNT: 03 TOTAL_DEFERRED_COUNT: 0 ENERGY: TOTAL_ENERGY_CONSUMPTION: 120050000.000 J4 MEM_USAGE: TOTAL_VRAM: 294912 MB USED_VRAM: 256000 MB5 THROTTLE: PPT_VIOLATION_STATUS: NOT ACTIVE6 SOCKET_THERMAL_VIOLATION_STATUS: NOT ACTIVE HBM_THERMAL_VIOLATION_STATUS: NOT ACTIVE- No edge sensor on MI35x. Use HOTSPOT. A tool that alarms on edge temperature reads the "unsupported" value (65535) and fires on healthy hardware; an easy mistake in a first set of health-check rules.
- PCIe replays: the host link retransmitting packets. Rising means a riser, retimer or slot problem.
- ECC totals, all 0. Per block:
-k. - Energy only goes up while the chip is powered: the best "is this data fresh?" signal (step 5).
- 256,000 MiB in use: the prefill engine pre-allocates its weights and KV cache. VRAM, not GFX%, is the reliable "someone owns this GPU" signal.
- Throttle status: why clocks are below maximum. PPT is the power cap; SOCKET_THERMAL and HBM_THERMAL are heat. NOT ACTIVE is good.
With no flags, metric prints everything (two screens). Narrow it: -u usage, -p power, -t temperature, -P PCIe, -e ECC totals, -k ECC per block, -m memory, -v throttling, -E energy, -x XGMI errors.
5 · Power bounces; energy tells the truth
$ amd-smi metric -g 4 -E -p # twice, 10 s apart## 10:00:00.000 UTC SOCKET_POWER: 520 W1 TOTAL_ENERGY_CONSUMPTION: 120000000.000 J## 10:00:10.000 UTC SOCKET_POWER: 400 W2 TOTAL_ENERGY_CONSUMPTION: 120004200.000 J3- First power reading: 520 W.
- Ten seconds later: 400 W. Each is an instant snapshot of a bursty load.
- The energy counter grew. Divide by the time between the reads to get the true average power.
Average power over those 10 seconds: (120004200.000 − 120000000.000) J ÷ 10.000 s = 4200.0 J ÷ 10.00 s ≈ 420 W. Neither snapshot (520 W, 400 W) was the average. Prometheus does the same division with rate(); GPU metrics follows this exact counter into a query.
6 · Who is using it? amd-smi process
$ amd-smi process -g 4GPU: 4 PROCESS_INFO: NAME: /usr/bin/python3.12 PID: 412341 VRAM_MEM: 2.0 MB PROCESS_INFO: NAME: /usr/bin/python3.12 PID: 415672 VRAM_MEM: 240.0 GB3 SDMA_USAGE: 123456789 us CU_OCCUPANCY: 0- The prefill driver process (
inference_api.prefill_driver.main): 2 MB of GPU memory. - The vLLM engine (
VLLM::EngineCore) that holds the memory. - 240.0 GB of the 256,000 MiB in use belong to this one process.
PIDs are host PIDs, not container PIDs. Two processes per GPU is normal for vLLM. Process names may need root. To get from these PIDs to the Pod, see GPU process ↔ Pod.
7 · The files under the tools: sysfs
$ cd /sys/class/drm/card5/device; cat …serial_number: SN-EXAMPLE-0005current_link_speed: 32.0 GT/s PCIecurrent_link_width: 16mem_info_vram_total: 3092376453121mem_info_vram_used: 2684354560002current_compute_partition: SPXhwmon/hwmon5/temp2_input: 50000 (temp2_label: junction)3hwmon/hwmon5/temp3_input: 38000 (temp3_label: mem)hwmon/hwmon5/power1_input: 3200000004ras/aca_umc: ue: 0 ce: 0 de: 0ras/event_state: Fatal Error: count:0, last_seqno:05- 309,237,645,312 B ÷ 1,048,576 = 294,912 MiB (amd-smi prints "294912 MB") = 288.0 GiB
- 268,435,456,000 B ÷ 1,048,576 = 256,000 MiB, 86.8 % of the card
- Millidegrees: 50000 ÷ 1000 = 50 °C junction; 38000 ÷ 1000 = 38 °C memory.
- Microwatts: 320,000,000 µW ÷ 1,000,000 = 320 W.
- No fatal events since boot.
amd-smi is a view over these files. They are cheap to read and keep working in scripts when the tools are slow. Units are raw: bytes, millidegrees, microwatts.
8 · Is it damaged? Are the links up?
$ amd-smi bad-pages -g 4GPU: 4 RETIRED: No bad pages found.1 PENDING: No bad pages found. UN_RES: No bad pages found.- Retired HBM pages, taken out of use after an uncorrectable error. Zero is normal. The list lives in an EEPROM on the board and survives reboots.
$ amd-smi xgmi -lXGMI LINK STATUS: bdf GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7GPU0 0000:11:00.0 SELF U U U U U U UGPU3 0000:6b:00.0 U U U SELF U U U UGPU4 0000:8b:00.0 U U U U SELF U U U1GPU7 0000:e5:00.0 U U U U U U U SELFLegend: U / D / X = Link is Up / Down / Disabled- Seven U: GPU 4 has a direct, up link to each of the other seven. A D means down: the GPU still works alone, but peer traffic and the hive reset are at risk.
GPU 4 is healthy: no damage, all links up. Pending pages are marked bad but not yet removed; unreservable (UN_RES) ones could not be removed.
Illustrative output with made-up values. Derived numbers are computed from the output shown.
Other subcommands
amd-smi --help on 26.0.0 lists: version, list, static, firmware, bad-pages, metric, process, event, topology, set, reset, monitor, xgmi, partition, ras, node, fabric. Beyond the walkthrough:
| Subcommand | What it shows (example) | Read it for |
|---|---|---|
firmware |
per-GPU firmware list, e.g. PSP_SOSDRV 00.00.00.10, PM 00.01.02.03, PLDM_BUNDLE 01.00.00.00 |
version drift across the 8 GPUs and across nodes after a repair |
topology -t / -o / -b |
XGMI between every pair, 1 hop, NUMA bandwidth 50000-50000 |
all 8 GPUs connect directly: one hive |
xgmi (no flags) |
bit rate 32 Gb/s, max bandwidth 512 Gb/s per link, read/write counters | a slow or erroring link; metric -x for errors |
partition |
NPS1, SPX, memory caps NPS1,NPS2 |
partition mode; other profiles (DPX, QPX, CPX) split a GPU into 2, 4 or 8 |
ras --cper, ras --afid --cper-file <f> |
error records and AMD Field IDs (needs sudo) |
evidence for AMD support; --cper --folder writes files |
node |
baseboard (UBB) temperatures, retimer regulators, node power threshold 12000 W, node power management disabled | the board around the GPUs |
Two firmware pieces matter most when things break. The PSP (Platform Security Processor) boots the chip and loads every other firmware. The SMU, shown as PM, manages power and clocks and executes resets. amd-smi firmware only reads; flashing goes through the BMC (Redfish) during provisioning or maintenance. Details of ras and the missing records for fatal ATHUB events are in AMD RAS and ECC.
rocm-smi and rocminfo
rocm-smi is the older tool, kept for scripts. With no flags it gives a compact table:
$ rocm-smi # example output, trimmed
Device Node IDs Temp Power Partitions SCLK MCLK Fan Perf PwrCap VRAM% GPU%
(DID, GUID) (Junction) (Socket) (Mem, Compute, ID)
0 2 0x75a3, 12345 45.0°C 250.0W NPS1, SPX, 0 100Mhz 2000Mhz 0% auto 1400.0W 0% 0%
4 6 0x75a3, 45678 50.0°C 320.0W NPS1, SPX, 0 2400Mhz 2000Mhz 0% auto 1400.0W 87% 0%
rocminfo lists agents: each CPU NUMA node first (two on the example node), then each GPU. A GPU in amd-smi list but missing from rocminfo points at the runtime layer (KFD, /dev/kfd permissions, a wedged device), not at the PCI link.
sysfs: the files under the tools
The full set used in this track (example values for GPU 4):
File (under /sys/class/drm/card5/device/) |
Example value | Meaning |
|---|---|---|
current_compute_partition, current_memory_partition |
SPX, NPS1 |
partition mode |
mem_info_vram_total, mem_info_vram_used |
309237645312, 268435456000 (bytes) |
HBM size and use |
gpu_busy_percent |
0 |
engine busy, % |
serial_number, product_name |
SN-EXAMPLE-0005, AMD Instinct MI355X OAM |
board identity |
current_link_speed, current_link_width |
32.0 GT/s PCIe, 16 |
PCIe link; fewer lanes or lower speed is a degraded link |
ras/features |
feature mask: 0xfffff |
which RAS blocks are on |
ras/aca_umc (also aca_gfx, aca_xgmi_wafl, …) |
ue: 0 ce: 0 de: 0 |
per-block error counts on this driver |
ras/event_state |
Fatal Error: count:0 |
fatal and poison events since boot |
ras/gpu_vram_bad_pages |
empty | bad pages, pfn : size : R/P/F |
xgmi_hive_info/xgmi_hive_id |
(number) | GPUs with the same id share a hive |
xgmi_physical_id |
6 |
position in the hive |
hwmon/hwmon*/temp2_input, temp3_input |
50000, 38000 |
junction, memory, in millidegrees |
hwmon/hwmon*/power1_input |
320000000 |
socket power, µW |
aer_dev_correctable, aer_dev_nonfatal, aer_dev_fatal |
all 0 |
PCIe Advanced Error Reporting counters |
The kernel documentation names the per-block files <block>_err_count. On recent amdgpu drivers (6.14.0 in this example) they do not exist; the counts are in ras/aca_<block> instead. Scripts that hard-code the old names silently read nothing.
The kernel log
journalctl -k is the most reliable record of a GPU fault. After a couple of weeks of uptime, dmesg can return no amdgpu lines at all (the ring buffer has wrapped), while journalctl -k still has the boot messages. Use journald.
journalctl -k --since -1d | grep -iE 'amdgpu.*(ras|reset|hang|timeout|fault|poison|fatal)'
A healthy boot shows one RAS line per GPU:
amdgpu 0000:e5:00.0: RAS INFO: ras initialized successfully, hardware ability[fffff] ras_mask[fffff]
amdgpu 0000:e5:00.0: AMDGPU RAS Is Ready.
Patterns that mean trouble:
| Pattern | Meaning |
|---|---|
RAS Intr triggered / ERREVENT_ATHUB_INTERRUPT |
a fatal hardware event; on a failing GPU it can repeat thousands of times |
GPU reset begin |
the driver started a reset; printed once |
ring … timeout |
a queue stopped making progress: a hang |
GPU reset succeeded, trying to resume |
only means the reinit path ran; not proof of recovery |
PSP resume failed / -62 |
the GPU’s boot processor did not come back; needs a cold power cycle |
bad page / poison |
memory errors being contained |
AER: |
PCIe errors on the link |
Validation and diagnostics (disruptive, described only)
These tools load the GPUs to the limit or push traffic between them. They need the node cordoned and drained, and the workload owner must confirm the GPUs are free first. They were not run for this page.
| Tool | What it does | Runtime | Notes |
|---|---|---|---|
RVS (ROCm Validation Suite, rvs) |
modules that stress compute, memory, PCIe and peer links, and check results against limits | minutes per module | public docs; config files choose the tests |
AGFHC (AMD GPU Field Health Check, /opt/amd/agfhc/agfhc) |
AMD’s field diagnostic, run before an RMA and for burn-in; adds multi-GPU XGMI tests | single_pass about 6 min; a burn-in recipe about 6 h |
distributed by AMD; no public docs found. Recipes include hbm_lvl5, pcie_lvl2, all_lvl5 |
| TransferBench | measures copy bandwidth between GPUs and host | minutes | finds a slow XGMI link or PCIe path |
Two limits to remember:
- A clean AGFHC pass does not prove a part is healthy. An intermittent fault that comes every 3 days on average is caught by a 2-hour test only about 3 % of the time.
- A passing burn-in is not a performance test. AGFHC checks health, not throughput.
Misconceptions
- “
amd-smi reset -G -g 3resets only GPU 3.” On MI35x it resets the hive, all 8 GPUs, and kills every process holding/dev/kfd. - “GPU index = slot.” No. Use
amd-smi list -eand the OAM ID. - “No lines in dmesg means no errors.” The ring buffer wraps. Read
journalctl -k. - “0 % GFX means the GPU is free.” Look at VRAM and
amd-smi process, then the claim.
Further reading
- AMD SMI CLI tool (reset scope, ras, monitor) · AMD SMI docs
- AMDGPU RAS support (sysfs and debugfs interfaces) · MI350 architecture
- ROCm Validation Suite · TransferBench · rocminfo · ROCm SMI library