Learn · GPU fundamentals
Reading GPU node topology: PCIe, NUMA and GPU links
A GPU server is not a flat box of 8 GPUs. Each GPU sits on a specific PCIe switch, under a specific CPU root complex, in a specific NUMA node, next to a specific NIC. Software that ignores this sends data the long way round. This page shows how to read the layout on a live node with read-only commands, using one 8-GPU MI355X node as the worked example. The concepts (PCIe, XGMI, NVLink, GPUDirect) are in memory and interconnects.
The idea first
A building analogy. The server is two buildings (the two CPU sockets) joined by a bridge (the socket-to-socket link). Each building is one NUMA node: a group of CPU cores with the RAM closest to them. Each building has four corridors (PCIe switches), and on each corridor sit one GPU and one NIC. Walking down your own corridor is quick. Going to another corridor means passing through the building’s central hall (the CPU’s I/O fabric); going to the other building means crossing the bridge as well.
| Piece | The question it answers |
|---|---|
| NUMA node | Which CPU cores and which RAM should feed this GPU? |
| PCIe root complex and switch | Which devices can talk without going through the CPU? |
| NIC on the same switch | Which network port should this GPU’s traffic use? |
| GPU-to-GPU links | Do GPUs talk directly, through a switch chip, or only through the CPU? |
| Link speed and width | Did every link come up at full speed? (A x16 link that trained as x8 halves bandwidth without any error.) |
Where the analogy breaks. The GPUs also have a private tunnel network of their own: on MI355X all 8 GPUs are wired directly to each other over XGMI, so GPU-to-GPU traffic never uses the corridors at all. And the numbers on the doors follow different schemes: GPU 0 is card1 and ens1np0, but renderD128 and rdma0 (below).
An 8-GPU MI355X node, mapped
An illustrative example node, gpu-node-1 (an 8-GPU OAM server: 2 CPU sockets, 8 × MI355X, 8 × 400 GbE RDMA NICs; all values below are made up but realistic). Each socket is one NUMA node, so GPUs 0–3 share NUMA node 0 and GPUs 4–7 share NUMA node 1. Pick a GPU and a NIC to see how far apart they are:
CPU 0 · NUMA node 0 · PCI buses 00–7f
NUMA 0cores 0–63, 128–191
root pci0000:00
PCIe Gen5 switch
GPU 00000:11:00.0
rdma0ens1np0
NUMA 0cores 0–63, 128–191
root pci0000:20
PCIe Gen5 switch
GPU 10000:2f:00.0
rdma1ens2np0
NUMA 0cores 0–63, 128–191
root pci0000:40
PCIe Gen5 switch
GPU 20000:4d:00.0
rdma2ens3np0
NUMA 0cores 0–63, 128–191
root pci0000:60
PCIe Gen5 switch
GPU 30000:6b:00.0
rdma3ens4np0
CPU 1 · NUMA node 1 · PCI buses 80–ff
NUMA 1cores 64–127, 192–255
root pci0000:80
PCIe Gen5 switch
GPU 40000:8b:00.0
rdma4ens5np0
NUMA 1cores 64–127, 192–255
root pci0000:a0
PCIe Gen5 switch
GPU 50000:a9:00.0
rdma5ens6np0
NUMA 1cores 64–127, 192–255
root pci0000:c0
PCIe Gen5 switch
GPU 60000:c7:00.0
rdma6ens7np0
NUMA 1cores 64–127, 192–255
root pci0000:e0
PCIe Gen5 switch
GPU 70000:e5:00.0
rdma7ens8np0
Socket-to-socket traffic crosses the CPUs' own interconnect. GPU-to-GPU traffic does not use this map at all: all 8 GPUs are one XGMI full mesh.
| amd-smi GPU | PCI address | NUMA node | Local CPU cores | PCIe root | NIC (same switch) | Net device | DRA device |
|---|---|---|---|---|---|---|---|
| 0 | 0000:11:00.0 | 0 | 0–63, 128–191 | pci0000:00 | rdma0 | ens1np0 | gpu-1-128 |
| 1 | 0000:2f:00.0 | 0 | 0–63, 128–191 | pci0000:20 | rdma1 | ens2np0 | gpu-2-129 |
| 2 | 0000:4d:00.0 | 0 | 0–63, 128–191 | pci0000:40 | rdma2 | ens3np0 | gpu-3-130 |
| 3 | 0000:6b:00.0 | 0 | 0–63, 128–191 | pci0000:60 | rdma3 | ens4np0 | gpu-4-131 |
| 4 | 0000:8b:00.0 | 1 | 64–127, 192–255 | pci0000:80 | rdma4 | ens5np0 | gpu-5-132 |
| 5 | 0000:a9:00.0 | 1 | 64–127, 192–255 | pci0000:a0 | rdma5 | ens6np0 | gpu-6-133 |
| 6 | 0000:c7:00.0 | 1 | 64–127, 192–255 | pci0000:c0 | rdma6 | ens7np0 | gpu-7-134 |
| 7 | 0000:e5:00.0 | 1 | 64–127, 192–255 | pci0000:e0 | rdma7 | ens8np0 | gpu-8-135 |
Things to notice:
- Numbering schemes differ. GPU index and
rdmaNcount from 0,ensNnp0andcardNfrom 1, render nodes from 128. On this example they happen to line up in the same order; on many real servers they do not (GPU 0 can sit next to the fifth NIC). Never assume “GPU 3 uses NIC 3”. Look it up, or let the scheduler do it from attributes. - The bus number tells the socket. Here buses
00–7fhang off CPU 0 and80–ffoff CPU 1. Larger servers may use a second PCI domain (0001:) for the second socket instead. - One slice per root complex. Each GPU and its NIC share one root complex and one PCIe Gen5 switch. Every GPU-NIC pair is
PIXdistance in NVIDIA terms: the best case. - The DRA device name encodes the card and render minor:
gpu-6-133iscard6andrenderD133.
Worked example: find GPU 5’s NIC from scratch
Suppose network traffic from the Pod on GPU 5 is slow and you want to check it uses the right NIC and CPU cores. The outputs below are illustrative example output for gpu-node-1; the common-prefix comparison is computed from them.
GPU 5: from index to NIC, cores and ResourceSlice
1 · Index → PCI address
$ amd-smi list # GPU 5 entry, trimmed
GPU: 5
BDF: 0000:a9:00.0
KFD_ID: 53412
NODE_ID: 7The BDF (bus:device.function, with the PCI domain in front) is the one name every tool agrees on. Indexes, card numbers and KFD node IDs all differ between tools; start from the BDF.
2 · PCI address → position in the PCIe tree
$ readlink -f /sys/bus/pci/devices/0000:a9:00.0
/sys/devices/pci0000:a0/0000:a0:01.1/0000:a8:00.0/0000:a9:00.0sysfs lays out each device under its parents. Read left to right: root complex pci0000:a0, the root port, then 0000:a8:00.0 (the switch’s upstream port), then the GPU itself. (Simplified: real paths also show a switch downstream port, and often bridges inside the GPU module, before the GPU.) The highlighted part is what step 3 finds in common with the NIC.
3 · Compare with every NIC: who shares the switch?
$ readlink -f /sys/class/infiniband/rdma5/device
/sys/devices/pci0000:a0/0000:a0:01.1/0000:a8:00.0/0000:aa:00.0| NIC | Path components shared with GPU 5 | Distance |
|---|---|---|
rdma0 | 0 | other CPU socket |
rdma1 | 0 | other CPU socket |
rdma2 | 0 | other CPU socket |
rdma3 | 0 | other CPU socket |
rdma4 | 0 | same socket, other root complex |
rdma5 | 3 | same PCIe switch |
rdma6 | 0 | same socket, other root complex |
rdma7 | 0 | same socket, other root complex |
Of the 8 NICs only rdma5 shares GPU 5’s root, root port and switch upstream port 0000:a8:00.0. Below that port the two sit on different downstream ports of that one switch: GPU-to-NIC traffic turns around inside the switch and never reaches the CPU.
4 · NIC → network interface and link state
$ ls /sys/class/infiniband/rdma5/device/net
ens6np0
$ cat /sys/class/infiniband/rdma5/ports/1/{link_layer,state,rate}
Ethernet
4: ACTIVE
400 Gb/sec (4X NDR)The RDMA device rdma5 is the network interface ens6np0: the name a Multus network attachment uses. It runs RoCE (RDMA over Ethernet), is up, at 400 Gb/s.
5 · NUMA node → CPU cores
$ cat /sys/bus/pci/devices/0000:a9:00.0/numa_node
1
$ lscpu | grep 'NUMA node1'
NUMA node1 CPU(s): 64-127,192-255The process feeding GPU 5 should run on these cores, and its pinned host buffers should live in NUMA node 1’s RAM. amd-smi static -g 5 reports the same affinity in its NUMA section.
6 · What Kubernetes sees
$ kubectl get resourceslice -o yaml # node `gpu-node-1`, trimmed
name: gpu-6-133 # card6, renderD133
attributes:
numaNode: {int: 1}
partitionProfile: {string: spx_nps1}
resource.kubernetes.io/pciBusID: {string: "0000:a9:00.0"}
resource.kubernetes.io/pcieRoot: {string: pci0000:a0}The AMD DRA driver publishes the same facts as device attributes. pcieRoot is a standard Kubernetes attribute name, so a claim can ask for a GPU and a NIC with matching roots, if the NIC is also published through DRA. When Pods get NICs through Multus instead, the pairing is by configuration, not by the scheduler. See GPU Pod networking.
All commands are read-only and safe on a node with live workloads.
The cost of getting it wrong, as packets:
- Its own NIC: same switchGPU 5 → switch →
rdma5. The CPU is not involved; this is the path GPUDirect RDMA wants. - Same socket, another sliceTo
rdma4: up to rootpci0000:a0, through CPU 1’s I/O fabric, down rootpci0000:80. Two extra PCIe hops, and the traffic competes with everything else on that CPU. - The other socketTo
rdma0: across the socket-to-socket link as well. The worst case (SYSin NVIDIA’s terms); NCCL’s default GPUDirect policy would not use this pair.
AMD: other views
PCIe tree
lspci -tv draws the same tree as the sysfs paths. GPU 5’s slice as illustrative output, trimmed and simplified (switch downstream ports folded into ...):
+-[0000:a0]-+-00.0 AMD Root Complex
| +-01.1-[a8-af]----00.0 PCIe Gen5 switch (upstream port a8:00.0)
| +-...-[a9]----00.0 AMD/ATI Instinct MI355X
| \-...-[aa]----00.0 Mellanox ConnectX-7
The bridge at a8:00.0 is the switch’s upstream port. The GPU and NIC are its children.
Link speed
$ cat /sys/bus/pci/devices/0000:11:00.0/{current_link_speed,current_link_width}
32.0 GT/s PCIe
16
Compare current_link_* with max_link_*. On a healthy node like this example, all GPUs and NICs run at 32 GT/s x16 (PCIe Gen5). ibv_devices, ibv_devinfo and rdma link show the NICs through the RDMA tools.
amd-smi
amd-smi listmaps GPU index to PCI address (BDF), UUID and KFD IDs.amd-smi static -g Nhas aNUMAsection (node and CPU affinity) and aBUSsection (PCIe generation, width, slot typeOAM).amd-smi topologyprints matrices between every GPU pair:ACCESS,WEIGHT,HOPS,LINK TYPE,NUMA BW,CACHE COHERENCY,ATOMICS,DMA,BI-DIRECTIONAL. In the example output every off-diagonal cell isXGMI, 1 hop, weight 15,72000-72000, coherent: a full mesh. A healthy 8-GPU MI355X node always looks like this; aPCIEcell or a missing GPU means a link failed. The output is shown in memory and interconnects.
The kernel’s view: KFD topology
The ROCm runtime builds its picture from /sys/class/kfd/kfd/topology/nodes/. Each CPU NUMA node and each GPU is a “node”; on the example node nodes 0–1 are CPUs and 2–9 are GPUs. Each GPU node has io_links (illustrative output for GPU 2):
nodes/4/io_links/0: type 2 node_from 4 node_to 0 weight 20 max_bandwidth 64000
nodes/4/io_links/1: type 11 node_from 4 node_to 2 weight 15 min_bandwidth 72000 max_bandwidth 72000
... (six more type 11 links, to nodes 3 and 5-9)
Link type 2 is PCIe (to CPU node 0, its NUMA node, 64,000 MB/s) and type 11 is XGMI (to each other GPU, 72,000 MB/s). The properties file of the same node carries hive_id, which is identical on all eight GPUs. The type numbers come from the kernel’s CRAT definitions; the output agrees with amd-smi topology.
NVIDIA: reading it yourself
The NVIDIA side below comes from NVIDIA documentation.
nvidia-smi topo -m prints a matrix of GPUs, NICs and CPU affinity. Each cell is a distance code. The tool prints its own legend; NVIDIA’s NCCL documentation defines the PCIe levels the same way:
| Code | Meaning |
|---|---|
X |
self |
NV# |
connected by NVLink, # links bonded |
PIX |
through at most one PCIe switch |
PXB |
through several PCIe switches, without the CPU |
PHB |
through a CPU PCIe host bridge, same NUMA node |
NODE |
between host bridges inside one NUMA node (from the tool’s legend; not checked in NCCL docs) |
SYS |
across the CPU-to-CPU interconnect between NUMA nodes |
On an NVSwitch system, GPU-to-GPU cells show NV#, and a well-built server shows PIX between each GPU and its paired NIC. The same codes are what NCCL_P2P_LEVEL and NCCL_NET_GDR_LEVEL take as thresholds.
Other read-only views:
nvidia-smi nvlinksubcommands show per-link state and counters (nvidia-smi nvlink -hlists them).nvidia-smi -qhas a Fabric section:Stateof the GPU’s handshake with Fabric Manager (Completed,In Progress,Not Started),Status, and on multi-node NVLink systems a Clique ID (GPUs that can reach each other over NVLink share one) and a Cluster UUID. On NVL72, every GPU in the rack’s NVLink domain should show the same cluster UUID. A GPU whose fabric state is notCompletedhas no usable NVLink.- The NVIDIA DRA driver publishes PCIe attributes per GPU and, for NVL72, NVLink domain information (see rack-scale GPUs); details are covered there.
Side by side
| Question | AMD command | NVIDIA command |
|---|---|---|
| List GPUs and PCI addresses | amd-smi list |
nvidia-smi -L, nvidia-smi -q |
| GPU-to-GPU links | amd-smi topology |
nvidia-smi topo -m, nvidia-smi nvlink |
| NUMA and CPU affinity | amd-smi static (NUMA), sysfs numa_node |
nvidia-smi topo -m (CPU affinity columns) |
| Fabric health | XGMI rows in amd-smi topology, amd-smi metric |
nvidia-smi -q Fabric section, Fabric Manager logs |
| PCIe tree and speed | lspci -tv, sysfs current_link_* |
same |
| RDMA NICs | ibv_devinfo, rdma link |
same |
How this shows up in operations
- Debugging slow GPU-to-network transfers: check the Pod’s GPU and NIC are in the same slice (same
pcieRootand NUMA node). A Pod given the “wrong” NIC sends every byte across the CPU interconnect. - After hardware work: compare
current_link_widthandamd-smi topologywith the table above. A reseated GPU that trained at x8, or a hive with a missing XGMI link, still “works”, just slower. - Naming in alerts: metrics and logs may name a GPU by index, PCI address, card number, render minor, KFD node or UUID. Keep a map like the one above per node type.