Learn · GPU fundamentals

Reading GPU node topology: PCIe, NUMA and GPU links

Updated 10 Nov 2026 · 8 min read

A GPU server is not a flat box of 8 GPUs. Each GPU sits on a specific PCIe switch, under a specific CPU root complex, in a specific NUMA node, next to a specific NIC. Software that ignores this sends data the long way round. This page shows how to read the layout on a live node with read-only commands, using one 8-GPU MI355X node as the worked example. The concepts (PCIe, XGMI, NVLink, GPUDirect) are in memory and interconnects.

The idea first

A building analogy. The server is two buildings (the two CPU sockets) joined by a bridge (the socket-to-socket link). Each building is one NUMA node: a group of CPU cores with the RAM closest to them. Each building has four corridors (PCIe switches), and on each corridor sit one GPU and one NIC. Walking down your own corridor is quick. Going to another corridor means passing through the building’s central hall (the CPU’s I/O fabric); going to the other building means crossing the bridge as well.

Piece The question it answers
NUMA node Which CPU cores and which RAM should feed this GPU?
PCIe root complex and switch Which devices can talk without going through the CPU?
NIC on the same switch Which network port should this GPU’s traffic use?
GPU-to-GPU links Do GPUs talk directly, through a switch chip, or only through the CPU?
Link speed and width Did every link come up at full speed? (A x16 link that trained as x8 halves bandwidth without any error.)

Where the analogy breaks. The GPUs also have a private tunnel network of their own: on MI355X all 8 GPUs are wired directly to each other over XGMI, so GPU-to-GPU traffic never uses the corridors at all. And the numbers on the doors follow different schemes: GPU 0 is card1 and ens1np0, but renderD128 and rdma0 (below).

An 8-GPU MI355X node, mapped

An illustrative example node, gpu-node-1 (an 8-GPU OAM server: 2 CPU sockets, 8 × MI355X, 8 × 400 GbE RDMA NICs; all values below are made up but realistic). Each socket is one NUMA node, so GPUs 0–3 share NUMA node 0 and GPUs 4–7 share NUMA node 1. Pick a GPU and a NIC to see how far apart they are:

Example MI355X server map (gpu-node-1)

CPU 0 · NUMA node 0 · PCI buses 00–7f

NUMA 0cores 0–63, 128–191

root pci0000:00

PCIe Gen5 switch

GPU 00000:11:00.0

rdma0ens1np0

NUMA 0cores 0–63, 128–191

root pci0000:20

PCIe Gen5 switch

GPU 10000:2f:00.0

rdma1ens2np0

NUMA 0cores 0–63, 128–191

root pci0000:40

PCIe Gen5 switch

GPU 20000:4d:00.0

rdma2ens3np0

NUMA 0cores 0–63, 128–191

root pci0000:60

PCIe Gen5 switch

GPU 30000:6b:00.0

rdma3ens4np0

CPU 1 · NUMA node 1 · PCI buses 80–ff

NUMA 1cores 64–127, 192–255

root pci0000:80

PCIe Gen5 switch

GPU 40000:8b:00.0

rdma4ens5np0

NUMA 1cores 64–127, 192–255

root pci0000:a0

PCIe Gen5 switch

GPU 50000:a9:00.0

rdma5ens6np0

NUMA 1cores 64–127, 192–255

root pci0000:c0

PCIe Gen5 switch

GPU 60000:c7:00.0

rdma6ens7np0

NUMA 1cores 64–127, 192–255

root pci0000:e0

PCIe Gen5 switch

GPU 70000:e5:00.0

rdma7ens8np0

Socket-to-socket traffic crosses the CPUs' own interconnect. GPU-to-GPU traffic does not use this map at all: all 8 GPUs are one XGMI full mesh.

amd-smi GPUPCI addressNUMA nodeLocal CPU coresPCIe rootNIC (same switch)Net deviceDRA device
00000:11:00.000–63, 128–191pci0000:00rdma0ens1np0gpu-1-128
10000:2f:00.000–63, 128–191pci0000:20rdma1ens2np0gpu-2-129
20000:4d:00.000–63, 128–191pci0000:40rdma2ens3np0gpu-3-130
30000:6b:00.000–63, 128–191pci0000:60rdma3ens4np0gpu-4-131
40000:8b:00.0164–127, 192–255pci0000:80rdma4ens5np0gpu-5-132
50000:a9:00.0164–127, 192–255pci0000:a0rdma5ens6np0gpu-6-133
60000:c7:00.0164–127, 192–255pci0000:c0rdma6ens7np0gpu-7-134
70000:e5:00.0164–127, 192–255pci0000:e0rdma7ens8np0gpu-8-135

Things to notice:

  • Numbering schemes differ. GPU index and rdmaN count from 0, ensNnp0 and cardN from 1, render nodes from 128. On this example they happen to line up in the same order; on many real servers they do not (GPU 0 can sit next to the fifth NIC). Never assume “GPU 3 uses NIC 3”. Look it up, or let the scheduler do it from attributes.
  • The bus number tells the socket. Here buses 00–7f hang off CPU 0 and 80–ff off CPU 1. Larger servers may use a second PCI domain (0001:) for the second socket instead.
  • One slice per root complex. Each GPU and its NIC share one root complex and one PCIe Gen5 switch. Every GPU-NIC pair is PIX distance in NVIDIA terms: the best case.
  • The DRA device name encodes the card and render minor: gpu-6-133 is card6 and renderD133.

Worked example: find GPU 5’s NIC from scratch

Suppose network traffic from the Pod on GPU 5 is slow and you want to check it uses the right NIC and CPU cores. The outputs below are illustrative example output for gpu-node-1; the common-prefix comparison is computed from them.

GPU 5: from index to NIC, cores and ResourceSlice

1 · Index → PCI address

$ amd-smi list        # GPU 5 entry, trimmed
GPU: 5
    BDF: 0000:a9:00.0
    KFD_ID: 53412
    NODE_ID: 7

The BDF (bus:device.function, with the PCI domain in front) is the one name every tool agrees on. Indexes, card numbers and KFD node IDs all differ between tools; start from the BDF.

2 · PCI address → position in the PCIe tree

$ readlink -f /sys/bus/pci/devices/0000:a9:00.0
/sys/devices/pci0000:a0/0000:a0:01.1/0000:a8:00.0/0000:a9:00.0

sysfs lays out each device under its parents. Read left to right: root complex pci0000:a0, the root port, then 0000:a8:00.0 (the switch’s upstream port), then the GPU itself. (Simplified: real paths also show a switch downstream port, and often bridges inside the GPU module, before the GPU.) The highlighted part is what step 3 finds in common with the NIC.

3 · Compare with every NIC: who shares the switch?

$ readlink -f /sys/class/infiniband/rdma5/device
/sys/devices/pci0000:a0/0000:a0:01.1/0000:a8:00.0/0000:aa:00.0
NICPath components shared with GPU 5Distance
rdma00other CPU socket
rdma10other CPU socket
rdma20other CPU socket
rdma30other CPU socket
rdma40same socket, other root complex
rdma53same PCIe switch
rdma60same socket, other root complex
rdma70same socket, other root complex

Of the 8 NICs only rdma5 shares GPU 5’s root, root port and switch upstream port 0000:a8:00.0. Below that port the two sit on different downstream ports of that one switch: GPU-to-NIC traffic turns around inside the switch and never reaches the CPU.

4 · NIC → network interface and link state

$ ls /sys/class/infiniband/rdma5/device/net
ens6np0
$ cat /sys/class/infiniband/rdma5/ports/1/{link_layer,state,rate}
Ethernet
4: ACTIVE
400 Gb/sec (4X NDR)

The RDMA device rdma5 is the network interface ens6np0: the name a Multus network attachment uses. It runs RoCE (RDMA over Ethernet), is up, at 400 Gb/s.

5 · NUMA node → CPU cores

$ cat /sys/bus/pci/devices/0000:a9:00.0/numa_node
1
$ lscpu | grep 'NUMA node1'
NUMA node1 CPU(s):      64-127,192-255

The process feeding GPU 5 should run on these cores, and its pinned host buffers should live in NUMA node 1’s RAM. amd-smi static -g 5 reports the same affinity in its NUMA section.

6 · What Kubernetes sees

$ kubectl get resourceslice -o yaml     # node `gpu-node-1`, trimmed
    name: gpu-6-133              # card6, renderD133
    attributes:
      numaNode: {int: 1}
      partitionProfile: {string: spx_nps1}
      resource.kubernetes.io/pciBusID: {string: "0000:a9:00.0"}
      resource.kubernetes.io/pcieRoot: {string: pci0000:a0}

The AMD DRA driver publishes the same facts as device attributes. pcieRoot is a standard Kubernetes attribute name, so a claim can ask for a GPU and a NIC with matching roots, if the NIC is also published through DRA. When Pods get NICs through Multus instead, the pairing is by configuration, not by the scheduler. See GPU Pod networking.

All commands are read-only and safe on a node with live workloads.

The cost of getting it wrong, as packets:

GPU 5 sending through three different NICs
CPU 1 (PCI BUSES 80–FF)CPU 0 (PCI BUSES 00–7F)CPU 1 I/O fabricNUMA 1Switch, rootpci0000:a0NUMA 1GPU 50000:a9:00.0rdma5ens6np0Switch, rootpci0000:80NUMA 1rdma4ens5np0CPU 0 I/O fabricNUMA 0Switch, rootpci0000:00NUMA 0rdma0ens1np0x16x16x16x16x16socket-to-socket linkx16x16
  1. Its own NIC: same switchGPU 5 → switch → rdma5. The CPU is not involved; this is the path GPUDirect RDMA wants.
  2. Same socket, another sliceTo rdma4: up to root pci0000:a0, through CPU 1’s I/O fabric, down root pci0000:80. Two extra PCIe hops, and the traffic competes with everything else on that CPU.
  3. The other socketTo rdma0: across the socket-to-socket link as well. The worst case (SYS in NVIDIA’s terms); NCCL’s default GPUDirect policy would not use this pair.
Part of an MI355X node. Only the first route stays inside one PCIe switch.

AMD: other views

PCIe tree

lspci -tv draws the same tree as the sysfs paths. GPU 5’s slice as illustrative output, trimmed and simplified (switch downstream ports folded into ...):

 +-[0000:a0]-+-00.0  AMD Root Complex
 |           +-01.1-[a8-af]----00.0  PCIe Gen5 switch (upstream port a8:00.0)
 |                               +-...-[a9]----00.0  AMD/ATI Instinct MI355X
 |                               \-...-[aa]----00.0  Mellanox ConnectX-7

The bridge at a8:00.0 is the switch’s upstream port. The GPU and NIC are its children.

$ cat /sys/bus/pci/devices/0000:11:00.0/{current_link_speed,current_link_width}
32.0 GT/s PCIe
16

Compare current_link_* with max_link_*. On a healthy node like this example, all GPUs and NICs run at 32 GT/s x16 (PCIe Gen5). ibv_devices, ibv_devinfo and rdma link show the NICs through the RDMA tools.

amd-smi

  • amd-smi list maps GPU index to PCI address (BDF), UUID and KFD IDs.
  • amd-smi static -g N has a NUMA section (node and CPU affinity) and a BUS section (PCIe generation, width, slot type OAM).
  • amd-smi topology prints matrices between every GPU pair: ACCESS, WEIGHT, HOPS, LINK TYPE, NUMA BW, CACHE COHERENCY, ATOMICS, DMA, BI-DIRECTIONAL. In the example output every off-diagonal cell is XGMI, 1 hop, weight 15, 72000-72000, coherent: a full mesh. A healthy 8-GPU MI355X node always looks like this; a PCIE cell or a missing GPU means a link failed. The output is shown in memory and interconnects.

The kernel’s view: KFD topology

The ROCm runtime builds its picture from /sys/class/kfd/kfd/topology/nodes/. Each CPU NUMA node and each GPU is a “node”; on the example node nodes 0–1 are CPUs and 2–9 are GPUs. Each GPU node has io_links (illustrative output for GPU 2):

nodes/4/io_links/0: type 2  node_from 4 node_to 0  weight 20 max_bandwidth 64000
nodes/4/io_links/1: type 11 node_from 4 node_to 2  weight 15 min_bandwidth 72000 max_bandwidth 72000
...                 (six more type 11 links, to nodes 3 and 5-9)

Link type 2 is PCIe (to CPU node 0, its NUMA node, 64,000 MB/s) and type 11 is XGMI (to each other GPU, 72,000 MB/s). The properties file of the same node carries hive_id, which is identical on all eight GPUs. The type numbers come from the kernel’s CRAT definitions; the output agrees with amd-smi topology.

NVIDIA: reading it yourself

The NVIDIA side below comes from NVIDIA documentation.

nvidia-smi topo -m prints a matrix of GPUs, NICs and CPU affinity. Each cell is a distance code. The tool prints its own legend; NVIDIA’s NCCL documentation defines the PCIe levels the same way:

Code Meaning
X self
NV# connected by NVLink, # links bonded
PIX through at most one PCIe switch
PXB through several PCIe switches, without the CPU
PHB through a CPU PCIe host bridge, same NUMA node
NODE between host bridges inside one NUMA node (from the tool’s legend; not checked in NCCL docs)
SYS across the CPU-to-CPU interconnect between NUMA nodes

On an NVSwitch system, GPU-to-GPU cells show NV#, and a well-built server shows PIX between each GPU and its paired NIC. The same codes are what NCCL_P2P_LEVEL and NCCL_NET_GDR_LEVEL take as thresholds.

Other read-only views:

  • nvidia-smi nvlink subcommands show per-link state and counters (nvidia-smi nvlink -h lists them).
  • nvidia-smi -q has a Fabric section: State of the GPU’s handshake with Fabric Manager (Completed, In Progress, Not Started), Status, and on multi-node NVLink systems a Clique ID (GPUs that can reach each other over NVLink share one) and a Cluster UUID. On NVL72, every GPU in the rack’s NVLink domain should show the same cluster UUID. A GPU whose fabric state is not Completed has no usable NVLink.
  • The NVIDIA DRA driver publishes PCIe attributes per GPU and, for NVL72, NVLink domain information (see rack-scale GPUs); details are covered there.

Side by side

Question AMD command NVIDIA command
List GPUs and PCI addresses amd-smi list nvidia-smi -L, nvidia-smi -q
GPU-to-GPU links amd-smi topology nvidia-smi topo -m, nvidia-smi nvlink
NUMA and CPU affinity amd-smi static (NUMA), sysfs numa_node nvidia-smi topo -m (CPU affinity columns)
Fabric health XGMI rows in amd-smi topology, amd-smi metric nvidia-smi -q Fabric section, Fabric Manager logs
PCIe tree and speed lspci -tv, sysfs current_link_* same
RDMA NICs ibv_devinfo, rdma link same

How this shows up in operations

  • Debugging slow GPU-to-network transfers: check the Pod’s GPU and NIC are in the same slice (same pcieRoot and NUMA node). A Pod given the “wrong” NIC sends every byte across the CPU interconnect.
  • After hardware work: compare current_link_width and amd-smi topology with the table above. A reseated GPU that trained at x8, or a hive with a missing XGMI link, still “works”, just slower.
  • Naming in alerts: metrics and logs may name a GPU by index, PCI address, card number, render minor, KFD node or UUID. Keep a map like the one above per node type.

Further reading