Learn · Operating GPU nodes
NVIDIA GPU tools: nvidia-smi, DCGM, Xid, Fabric Manager, IMEX
This page lists the standard NVIDIA tools, checked against NVIDIA’s documentation, next to their AMD counterparts. The NVIDIA commands were not run for this page; re-check versions and paths on your own nodes.
The idea
The questions do not change with the vendor: which GPU is which, how is it doing, is it damaged, who uses it, are its links up? Only the tool names do. nvidia-smi plays the part of amd-smi; DCGM (Data Center GPU Manager) adds a resident service that watches health and runs diagnostics; and NVIDIA’s driver writes numbered Xid errors to the kernel log, each with a recommended action, where AMD writes free-text lines.
One piece has no AMD counterpart. On HGX and DGX boards the GPUs do not connect to each other directly: they connect through NVSwitch chips, like servers through a network switch. A switch needs its routes programmed before traffic flows, and Fabric Manager is the service that programs them, like a controller in a software-defined network. Where the analogy breaks: a server with an unconfigured switch still works on its own; a CUDA program on an HGX node without a running Fabric Manager does not start at all, even though nvidia-smi lists every GPU.
AMD and NVIDIA, task by task
Pick a task and a vendor. AMD output is illustrative example output (an example MI355X node, GPU 4). NVIDIA commands are from NVIDIA’s documentation and were not run for this page, so no output is shown for them.
$ amd-smi list -e -g 4GPU: 4 BDF: 0000:8b:00.0 RENDER: renderD132 CARD: card5 OAM_ID: 6Index, PCI BDF, render node, DRM card, OAM slot.
$ amd-smi monitor -p -t -u -m -eGPU XCP POWER PWR_CAP GPU_T MEM_T GFX_CLK GFX% MEM% SINGLE_ECC DOUBLE_ECC PCIE_REPLAY 4 0 400 W 1400 W 50 °C 38 °C 2400 MHz 12 % 5 % 0 0 0Power against the cap, temperatures, clock, ECC and PCIe replays: one line per GPU.
$ amd-smi metric -g 4 -eGPU: 4 ECC: TOTAL_CORRECTABLE_COUNT: 0 TOTAL_UNCORRECTABLE_COUNT: 0 TOTAL_DEFERRED_COUNT: 0Correctable, uncorrectable and deferred totals; -k gives them per block.
$ amd-smi bad-pages -g 4GPU: 4 RETIRED: No bad pages found. PENDING: No bad pages found. UN_RES: No bad pages found.Retired HBM pages; 128 is the budget.
$ amd-smi xgmi -lXGMI LINK STATUS: bdf GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7GPU4 0000:8b:00.0 U U U U SELF U U U U / D / X = Link is Up / Down / DisabledSeven U per row: every XGMI peer link is up.
$ amd-smi process -g 4GPU: 4 PROCESS_INFO: PID: 41234 VRAM_MEM: 2.0 MB PROCESS_INFO: PID: 41567 VRAM_MEM: 240.0 GBHost PIDs and their GPU memory.
$ amd-smi metric -g 4 -vGPU: 4 THROTTLE: PPT_VIOLATION_STATUS: NOT ACTIVE SOCKET_THERMAL_VIOLATION_STATUS: NOT ACTIVE HBM_THERMAL_VIOLATION_STATUS: NOT ACTIVEWhich limit holds the clock down: power cap (PPT), socket or HBM heat.
$ agfhc …, rvs …, TransferBench … # disruptive(disruptive: described only)AGFHC single_pass (minutes) or burn-in (hours); RVS modules; TransferBench for link bandwidth.
$ amd-smi reset -G -g 4 # never on a serving node(not run: resets all 8 GPUs of the hive)On MI35x a GPU reset is a hive reset, and it kills every /dev/kfd user.
$ nvidia-smi -L$ nvidia-smi --query-gpu=index,uuid,pci.bus_id --format=csv$ dcgmi discovery -lIndex, UUID and PCI bus ID per GPU; DCGM also lists NVSwitches.
$ nvidia-smi$ nvidia-smi --query-gpu=index,temperature.gpu,power.draw,memory.used,utilization.gpu --format=csvThe summary table, or chosen fields for scripts. Check field names with nvidia-smi --help-query-gpu first: they change between drivers.
$ nvidia-smi -q -d ECC Volatile (since driver load) and aggregate (lifetime) single- and double-bit counts.
$ nvidia-smi -q -d ROW_REMAPPER Remapped HBM rows: correctable, uncorrectable, pending, and whether a remap failed (Xid 64).
$ nvidia-smi nvlink -s$ nvidia-smi nvlink -e$ nvidia-smi topo -mNVLink status and error counters; topo -m is the GPU/NIC connection matrix with CPU and NUMA affinity.
$ nvidia-smi --query-compute-apps=pid,gpu_uuid,used_memory --format=csv Host PIDs per GPU UUID, with memory used.
$ nvidia-smi -q -d PERFORMANCE Clock event reasons (formerly "throttle reasons"): power cap, thermal, sync boost, and so on.
$ dcgmi diag -r 3 Under 35 min on 8 Hopper GPUs: stress, NVBandwidth, NCCL tests, targeted power. Level 2 and above are disruptive.
$ nvidia-smi -r -i 4 One GPU on HGX (needs root and no process on it). On NVL72 a reset re-registers the GPU with the fabric.
The start order on an NVSwitch node
- 1 · Driver firstThe kernel driver and the GSP firmware load.
nvidia-smialready lists all GPUs at this point. - 2 · Fabric Manager checks versionsAt start, Fabric Manager checks the driver stack and aborts if the versions are incompatible. In practice, keep the Fabric Manager version equal to the driver version.
- 3 · Routes programmedFabric Manager (with NVLSM on B200/B300) programs the NVSwitch routes. Now GPU-to-GPU traffic can flow.
- 4 · CUDA startsOnly now does CUDA initialization succeed.
- Fabric Manager missing or mismatchedA driver upgrade without the matching Fabric Manager: every GPU is listed, none is usable. CUDA init fails with
cudaErrorSystemNotReady. Stopping Fabric Manager also loses fabric state, and GPUs need a reset after it restarts.
Worked example: five Xids, five decisions
When the driver logs NVRM: Xid in journalctl -k, the line carries the PCI address and the code. The code alone decides the first move. Descriptions and immediate actions below are quoted from NVIDIA’s Xid catalog; the last line of each step is a suggested response.
From an Xid code to an action
1 · Xid 13: Graphics Engine Exception
catalog action: RESTART_APP AMD analogue: a job error, no RAS event
Usually a bad kernel or pointer. The workload owner restarts the replica; hold the node only if it repeats across workloads.
2 · Xid 63: GPU memory remapping event
catalog action: IGNORE AMD analogue: an HBM page retired, under the budget
Graceful HBM wear. Keep serving; ticket it and watch ROW_REMAPPER for pending or failed remaps.
3 · Xid 64: GPU memory remapping failure
catalog action: RESET_GPU AMD analogue: bad-page threshold exceeded
Hold, drain, nvidia-bug-report.sh, then reset the GPU; contact NVIDIA support.
4 · Xid 79: GPU has fallen off the bus
catalog action: RESTART_BM AMD analogue: GPU missing from amd-smi list / lspci
Hold, drain, capture (nvidia-bug-report.sh), then restart the bare-metal host; reseat or RMA if it repeats.
5 · Xid 119: GSP RPC Timeout
catalog action: RESET_GPU AMD analogue: PSP/SMU firmware not answering
The GPU's own processor stopped answering. Hold, drain, capture, reset the GPU (one GPU on HGX, not the baseboard).
Catalog text quoted from NVIDIA's Xid catalog. Response lines follow a hold → drain → capture → recover order.
The pattern: RESTART_APP codes are usually the program’s fault, IGNORE codes are bookkeeping, and RESET_GPU / RESTART_BM codes are the hardware’s. On HGX a reset is one GPU; on MI35x it is the whole hive.
The pieces on an NVIDIA node
| Piece | What it is | AMD equivalent |
|---|---|---|
| NVIDIA kernel driver + GSP firmware | the driver; recent GPUs offload much of it to the GPU System Processor | amdgpu + PSP/SMU firmware |
NVML + nvidia-smi |
management library and its CLI | AMD SMI library + amd-smi |
DCGM (nv-hostengine, dcgmi) |
Data Center GPU Manager: health watches, diagnostics, telemetry | no single equivalent: amd-smi + RVS/AGFHC + exporter |
| dcgm-exporter | Prometheus exporter built on DCGM | device-metrics-exporter |
Fabric Manager (nvidia-fabricmanager) |
configures NVSwitch routing on HGX/DGX boards | none: XGMI needs no fabric manager |
| NVLSM | NVLink subnet manager, a Fabric Manager dependency on B200/B300 | none |
IMEX (nvidia-imex) |
lets GPUs on different trays of one NVLink domain share memory | none on MI355X |
nvidia-smi: safe, read-only
| Command | Shows | AMD equivalent |
|---|---|---|
nvidia-smi |
summary: GPUs, utilization, memory, processes | amd-smi monitor -q |
nvidia-smi -L |
GPU list with UUIDs | amd-smi list |
nvidia-smi -q |
the full record for each GPU | amd-smi static + metric |
nvidia-smi -q -d ECC |
ECC counters, volatile and aggregate | amd-smi metric -e -k |
nvidia-smi -q -d ROW_REMAPPER |
remapped HBM rows, pending and failed | amd-smi bad-pages |
nvidia-smi -q -d PERFORMANCE |
clock event (throttle) reasons | amd-smi metric -v |
nvidia-smi -q -d TEMPERATURE / POWER / CLOCK |
sensors | amd-smi metric -t -p -c |
nvidia-smi --query-gpu=<fields> --format=csv |
chosen fields, for scripts | amd-smi metric --csv |
nvidia-smi --query-compute-apps=pid,gpu_uuid,used_memory --format=csv |
processes per GPU | amd-smi process |
nvidia-smi topo -m |
GPU/NIC connection matrix with CPU and NUMA affinity | amd-smi topology |
nvidia-smi nvlink -s / -e |
NVLink status / error counters | amd-smi xgmi / metric -x |
nvidia-smi mig -lgi |
MIG GPU instances | amd-smi partition |
The -d filter accepts: MEMORY, UTILIZATION, ECC, TEMPERATURE, POWER, CLOCK, COMPUTE, PIDS, PERFORMANCE, SUPPORTED_CLOCKS, PAGE_RETIREMENT, ACCOUNTING, ENCODER_STATS, SUPPORTED_GPU_TARGET_TEMP, VOLTAGE, FBC_STATS, ROW_REMAPPER, GSP_FIRMWARE_VERSION, POWER_SMOOTHING, POWER_PROFILES, BANK_REMAPPER (from the nvidia-smi documentation).
Query fields. --query-gpu takes field names such as index, uuid, pci.bus_id, utilization.gpu, memory.used, temperature.gpu and power.draw. The manual page does not list them; nvidia-smi --help-query-gpu does, for the installed driver. Field names change between drivers. One example: the throttle fields were renamed to clocks_event_reasons, and the old clocks_throttle_reasons names came back only for compatibility. Always check --help-query-gpu on the node before you script a field.
# A morning check on an HGX node (not run for this page)
nvidia-smi --query-gpu=index,pci.bus_id,temperature.gpu,power.draw,memory.used,utilization.gpu --format=csv
nvidia-smi -q -d ECC,ROW_REMAPPER | grep -iE 'pending|failure|uncorr'
DCGM
DCGM runs as a host service (nv-hostengine) or inside the GPU operator. Most NVIDIA tooling, including dcgm-exporter and NVSentinel, talks to it.
| Command | Use |
|---|---|
dcgmi discovery -l |
GPUs (and NVSwitches) DCGM sees |
dcgmi health -s <systems> then dcgmi health -c |
set background health watches (PCIe, memory, NVLink, thermal…), then check them |
dcgmi dmon -e <field ids> |
stream chosen fields |
dcgmi diag -r 1 … -r 4 |
diagnostics; each level includes the ones below |
Diagnostic levels, with NVIDIA’s measured times on Hopper systems:
| Level | Name | 8-GPU duration | Adds |
|---|---|---|---|
-r 1 |
Short | under 2.5 s | software and deployment checks |
-r 2 |
Medium | under 10.5 min | more hardware plugins (see NVIDIA’s table for the exact list) |
-r 3 |
Long | under 35 min | stress, NVBandwidth, NCCL tests, targeted power |
-r 4 |
Extra long | under 2.25 h | memtest and pulse test |
The deployment check fails if other graphics processes run on the GPU, and the GPU memory test takes 75 % of GPU memory by default. Treat level 2 and above as disruptive. dcgmi diag -r 3 is the closest NVIDIA counterpart of AMD’s AGFHC.
Xid: NVIDIA’s error codes
The NVIDIA driver logs Xid errors to the kernel log:
journalctl -k | grep -i 'NVRM: Xid'
The line carries the PCI address and the code. NVIDIA’s Xid catalog assigns each code an immediate action (a “resolution bucket”) and an investigatory action. The codes most likely to matter on a serving fleet, quoted from the catalog:
| Xid | Catalog description | Immediate action |
|---|---|---|
| 13 | Graphics Engine Exception | RESTART_APP |
| 31 | GPU memory page fault | RESTART_APP |
| 43 | GPU stopped processing | IGNORE |
| 48 | Double Bit ECC Error | WORKFLOW_XID_48 |
| 62 | Internal micro-controller halt | RESET_GPU |
| 63 | GPU memory remapping event | IGNORE |
| 64 | GPU memory remapping failure | RESET_GPU |
| 74 | NVLINK Error | WORKFLOW_NVLINK_ERR |
| 79 | GPU has fallen off the bus | RESTART_BM (restart the bare-metal host) |
| 92 | High single-bit ECC error rate | IGNORE |
| 94 | Contained memory error | RESTART_APP |
| 95 | Uncontained memory error | RESET_GPU |
| 109 | Context Switch Timeout Error | RESET_GPU |
| 119 / 120 | GSP RPC Timeout / GSP Error | RESET_GPU |
| 136 | Link Training Failed | RESET_GPU |
| 140 | ECC Unrecovered Error | RESET_GPU |
| 143 | GPU Initialization Error | RESET_GPU |
| 145 | NVLINK: RLW Error | WORKFLOW_NVLINK5_ERR |
| 154 | GPU Recovery Action Changed | informational, about another Xid |
How to read it: an application bug and a hardware fault can both produce an Xid. 13 and 31 usually mean a bad kernel or bad pointer, so restart the app. 79, 95, 119 and 140 mean the GPU itself needs a reset or the host a restart. On HGX, a reset is per GPU. On MI35x it is the whole hive.
NVLink systems: Fabric Manager, NVLSM, IMEX
Fabric Manager configures the NVSwitch chips that connect GPUs on HGX/DGX baseboards. Facts from NVIDIA’s Fabric Manager guide:
- Service
nvidia-fabricmanager; log/var/log/fabricmanager.log, orjournalctl -u nvidia-fabricmanager. - At start it checks the driver stack and aborts if the versions are incompatible. Keep the Fabric Manager version equal to the driver version.
- If a CUDA application starts before Fabric Manager finishes, or Fabric Manager fails, CUDA init fails with
cudaErrorSystemNotReady. Stopping Fabric Manager loses fabric state, and GPUs need a reset after it restarts. - On B200/B300 it depends on NVLSM (log
/var/log/opensm.log).
The ops lesson: on an HGX node, a driver upgrade without a matching Fabric Manager leaves every GPU unusable, even though nvidia-smi lists them all.
IMEX (nvidia-imex) supports GPU memory export and import across nodes in one NVLink domain (NVL72: 18 compute trays). Nodes in an IMEX domain find each other through nodes_config.cfg. NVIDIA’s DRA driver manages IMEX for Kubernetes workloads through a ComputeDomain object. Rack-scale specifics are in Rack-scale GPUs.
NVSentinel
NVSentinel is NVIDIA’s open-source fault agent for Kubernetes GPU nodes: roughly node-problem-detector plus remediation, packaged for NVIDIA GPUs. From its README (v1.31.0):
- Detect: GPU, NIC and system faults from DCGM, syslog and cloud maintenance events, reported as node conditions. It talks to DCGM directly, so the GPU operator must run DCGM as a standalone service.
- Protect: optional quarantine (cordon) and drain.
- Remediate: optional GPU reset or reboot, then return to service.
- A circuit breaker pauses quarantine once the nodes cordoned in 5 minutes reach 50 % of the cluster.
- The default install only monitors. Each further step is a separate Helm flag. It needs Kubernetes 1.34+ and has a flag for GPU operator DRA mode.
Starting with detection only is the cautious path: automatic drain and reboot must agree with whoever owns draining your workloads.
Evidence for NVIDIA support
nvidia-bug-report.sh (ships with the driver; run as root) collects kernel logs, nvidia-smi -q, PCI state and driver details into one compressed log for NVIDIA support. Run it before any reset or reboot, for the same reason as on AMD: recovery destroys evidence.
AMD and NVIDIA side by side
| Task | AMD MI35x | NVIDIA |
|---|---|---|
| Inventory | amd-smi list -e |
nvidia-smi -L, dcgmi discovery -l |
| Health now | amd-smi metric -e -k, journalctl -k |
dcgmi health -c, Xid in journalctl -k |
| Memory wear | amd-smi bad-pages |
nvidia-smi -q -d ROW_REMAPPER |
| Links | amd-smi xgmi |
nvidia-smi nvlink -s |
| Stress test (drained) | AGFHC, RVS, TransferBench | dcgmi diag -r 3, nvbandwidth |
| Reset scope | whole hive (8 GPUs) | one GPU (HGX); NVLink-domain effects on NVL72 |
| Fabric service | none | Fabric Manager (+ NVLSM, IMEX) |
| Node fault agent | node-problem-detector with AMD rules | NVSentinel |