Learn · GPUs in Kubernetes
GPU Pod networking: Multus, macvlan, whereabouts and RDMA devices
A normal Pod has one interface, eth0, from the cluster CNI (Cilium in the sample cluster on this page). That network is built for services and ordinary Pod traffic, not for moving gigabytes per second between GPUs on different servers. An LLM prefill Pod that hands its KV cache to a remote decode server (or any Pod doing multi-node collectives) needs two more things: an IP interface on the data network (the 400G Ethernet fabric), and the RDMA device files of the node’s ConnectX-7 NICs, so a transfer library can write data over RoCE straight to the far side.
The idea
A prefill Pod has two very different jobs for the network. It must talk to the rest of the cluster (API calls, metrics, health checks), and it must ship its KV cache, gigabytes per request burst, to a remote peer as fast as the hardware allows. One road cannot do both well, so it gets two, plus a key to the fast vehicles:
| Piece | Everyday picture | The question it answers |
|---|---|---|
eth0 (Cilium) |
the city streets: go anywhere, traffic lights and police (policy) | “How does this Pod reach Services and other Pods?” |
net1 (Multus + macvlan) |
a private on-ramp to the freight highway (the 400G data fabric) | “Which data NIC, and which IP on the data network?” |
rdma/hca_devices (device plugin) |
the keys to the highway’s trucks: device files that let the process drive the NIC directly | “May this container use RDMA at all?” |
| GPU and its NIC under one PCIe switch | the loading dock next to the warehouse | “How far do the bytes travel inside the server?” |
Where the picture breaks: RDMA traffic does not use net1’s IP at all (the NIC uses its own addresses, see below), and the keys open every truck on the node, not one.
- eth0Cilium, through Multus, gives the Pod its ordinary interface.
- net1macvlan makes a child of one 400G NIC inside the Pod; whereabouts gives it a data-network address.
- RDMA keysThe RDMA shared device plugin puts
/dev/infiniband/*into the container.
The parts
| Part | What it does | In the sample cluster |
|---|---|---|
| Cilium | primary Pod network (eth0), NetworkPolicy, services |
DaemonSet kube-system/cilium |
| Multus | meta-CNI: calls Cilium for eth0, then adds extra interfaces listed in a Pod annotation |
DaemonSet kube-system/multus (Multus v4) |
| NetworkAttachmentDefinition (NAD) | names one extra network | gpu-demo/data-net-1 … data-net-8 |
| macvlan | CNI plugin that creates a child interface with its own MAC on a host NIC | from /opt/cni/bin/macvlan |
| whereabouts | cluster-wide IPAM for the extra networks, state in CRDs | DaemonSet kube-system/whereabouts |
| RDMA shared device plugin | device plugin that hands RDMA device files to containers | DaemonSet kube-system/rdma-device-plugin, image k8s-rdma-shared-dev-plugin, resource rdma/hca_devices |
All of these typically run on every node in the cluster, not only on GPU nodes.
The NICs on a GPU node
The example 8-GPU OAM server has eight ConnectX-7 400 GbE NICs, one per GPU. Illustrative output on gpu-node-1:
$ rdma link
link rdma0/1 state ACTIVE physical_state LINK_UP netdev ens1np0
link rdma1/1 state ACTIVE physical_state LINK_UP netdev ens2np0
… # 8 links, all ACTIVE
| NUMA | NIC (RDMA) | NIC PCI | GPU | GPU PCI | Shared PCIe switch |
|---|---|---|---|---|---|
| 0 | ens1np0 (rdma0) | 0000:12:00.0 | gpu-1-128 | 0000:11:00.0 | 0000:10:00.0 |
| 0 | ens2np0 (rdma1) | 0000:30:00.0 | gpu-2-129 | 0000:2f:00.0 | 0000:2e:00.0 |
| 0 | ens3np0 (rdma2) | 0000:4e:00.0 | gpu-3-130 | 0000:4d:00.0 | 0000:4c:00.0 |
| 0 | ens4np0 (rdma3) | 0000:6c:00.0 | gpu-4-131 | 0000:6b:00.0 | 0000:6a:00.0 |
| 1 | ens5np0 (rdma4) | 0000:8c:00.0 | gpu-5-132 | 0000:8b:00.0 | 0000:8a:00.0 |
| 1 | ens6np0 (rdma5) | 0000:aa:00.0 | gpu-6-133 | 0000:a9:00.0 | 0000:a8:00.0 |
| 1 | ens7np0 (rdma6) | 0000:c8:00.0 | gpu-7-134 | 0000:c7:00.0 | 0000:c6:00.0 |
| 1 | ens8np0 (rdma7) | 0000:e6:00.0 | gpu-8-135 | 0000:e5:00.0 | 0000:e4:00.0 |
All 8 NICs pair with one GPU each under one PCIe Gen 5 switch, both on x16 links at 32 GT/s (from the sysfs device paths; see node topology). Traffic between a GPU and “its” NIC stays under that switch; traffic to any other NIC climbs to the CPU, and for the four on the other socket crosses the socket-to-socket link too.
Worked example: one prefill Pod gets its networks
Pod llm-prefill-5f7d8c9b6-p4x7k on gpu-node-1. Illustrative example: names, times and config are made up but realistic; IP addresses are placeholders.
From sandbox to eth0, net1 and RDMA in one to two seconds
1 · The Pod asks for three different things
Three mechanisms, none of which knows about the others. The scheduler has already placed the Pod and DRA has picked gpu-5-132, which sits on NUMA node 1. Networking happens next, on the node.
metadata:
annotations:
k8s.v1.cni.cncf.io/networks: gpu-demo/data-net-5 # → Multus: a second interface
spec:
containers:
- name: prefill
resources:
claims: [{name: gpu}] # → DRA: one GPU
limits: {cpu: "8", memory: 257Gi, rdma/hca_devices: "1"} # → device plugin: RDMA files2 · t = 0 s: containerd creates the sandbox and calls Multus
// /etc/cni/net.d/00-multus.conf (sorts first, so containerd runs it)
{ "type": "multus", "cniConf": "/host/etc/cni/multus/net.d",
"readinessindicatorfile": "/etc/cni/net.d/05-cilium.conflist",
"kubeconfig": "/etc/cni/net.d/multus.d/multus.kubeconfig",
"delegates": [{"name": "cilium", "plugins": [{"type": "cilium-cni"}]}] }"type": "multus" is thin mode: the multus binary runs once per Pod as a CNI plugin and reads the API with the host’s kubeconfig (thick mode uses multus-shim plus a daemon). readinessindicatorfile makes it refuse to run until Cilium’s config exists, so no Pod starts without a primary network. The journal shows CmdAdd … eth0 for this Pod in the same second.
3 · Cilium builds eth0
Multus calls its delegate, Cilium: a veth pair, a Pod IP from Cilium’s cluster pool, eBPF policy. Traffic to other nodes leaves through a VXLAN tunnel (routing-mode: tunnel, tunnel-protocol: vxlan in cilium-config) over the host’s default route, eno1 (25 GbE). This is the road for API calls, metrics and service traffic.
4 · The NAD is only a name; a node file says which NIC
# kubectl -n gpu-demo get net-attach-def data-net-5 -o yaml
spec:
config: "" # empty: look for a file of the same name on the node// /etc/cni/multus/net.d/data-net-5.conf on gpu-node-1 (addresses are placeholders)
{ "name": "data-net-5", "type": "macvlan", "master": "ens5np0", "mode": "bridge",
"ipam": { "type": "whereabouts", "range": "<data-subnet>/22", "range_start": "<first>", "range_end": "<last>",
"routes": [{"dst": "<data-subnet>/22", "gw": "0.0.0.0"}, {"dst": "<data-supernet>/16", "gw": "<leaf-gateway>"}] } }With an empty config, Multus reads the file whose name matches. The node provisioning writes one per NIC on every node: data-net-N → ensNnp0. One cluster-wide NAD therefore works on nodes with different subnets.
5 · macvlan makes net1, whereabouts gives it an address
The macvlan plugin creates a child of ens5np0 with its own MAC (02:00:…), moves it into the Pod as net1, mode bridge (children of one parent talk directly). whereabouts picks a free address from the /22 and records it in an IPPool CR in kube-system, so two nodes never hand out the same one. Its reconciler ("reconciler_cron_expression": "30 4 * * *", daily 04:30) frees addresses of Pods that vanished without a CNI DEL.
Properties to remember: the host cannot reach its own macvlan children through the parent; the switch port sees one MAC per Pod; Cilium does not see net1, so NetworkPolicy does not apply to it.
6 · Multus writes the result on the Pod
// k8s.v1.cni.cncf.io/network-status (addresses are placeholders)
[{"name": "cilium", "interface": "eth0", "ips": ["<pod-ip>"], "mac": "<pod-mac>", "default": true, "gateway": ["<gw>"]},
{"name": "gpu-demo/data-net-5", "interface": "net1", "ips": ["<data-net-ip>"], "mac": "<net1-mac>"}]If any plugin fails, sandbox creation fails: the Pod stays ContainerCreating with a FailedCreatePodSandBox event naming the plugin.
7 · The RDMA device files go in, the container starts (t ≈ 2 s)
rdma/hca_devices: 1 → Allocate response (kubelet device-plugin checkpoint):
/dev/infiniband/uverbs0 … uverbs7 umad0 … umad7 rdma_cmThe kubelet gives the container the device files the RDMA shared device plugin returned: all 8 NICs, not one. The init container ran at about t = 1 s, prefill started at about t = 2 s. From sandbox to running took one to two seconds; loading the model then took 4 min 0 s.
Illustrative config and annotations; whereabouts and macvlan internals per their docs.
Two roads out of the Pod
The same Pod, with the bytes moving. Step through: the first road is ordinary IP traffic; the last two are how a block of GPU memory (for example KV cache) leaves the GPU.
- eth0: ordinary trafficAn HTTP call or a metrics scrape: through the veth, Cilium’s eBPF programs, a VXLAN tunnel and the 25 GbE port. Policy applies.
- net1: an IP on the data networkIP traffic on the 400G data network, through the macvlan child of
ens5np0. Cilium is not in the path. - RDMA: the CPU only posts workTo send a KV block, the library posts a work request to a queue the NIC reads (through
/dev/infiniband/uverbs*) and rings the NIC’s doorbell. No socket, no kernel copy. - RDMA: the NIC moves the bytesWith GPU-direct RDMA the NIC reads the KV block straight from HBM, through the shared PCIe switch, and writes it to the far side over RoCE v2. Host RAM and the CPU are not on the data path. Without it, the block is first copied to host RAM. See GPU memory and interconnects.
On this Pod, net1 sits on ens5np0, the NIC under the same switch as its GPU. That is luck of the draw: the NAD is fixed in the Deployment template (every Pod of this Deployment uses data-net-5, a second Deployment uses data-net-6), while DRA picks the GPU. A Pod of the second Deployment on the same node holds gpu-1-128 (NUMA 0) and its net1 is on ens6np0 (NUMA 1, the other socket). Try other pairs:
How the RDMA devices get in
rdma/hca_devices is the device plugin model from GPU allocation in k8s. The plugin’s config, from ConfigMap kube-system/rdma-devices:
{
"periodicUpdateInterval": 300,
"configList": [{
"resourceName": "hca_devices",
"rdmaHcaMax": 1000,
"selectors": {"vendors": ["15b3", "14e4"], "drivers": ["mlx5_core", "bnxt_en"]}
}]
}
- Selectors: PCI vendor
15b3(NVIDIA/Mellanox) or14e4(Broadcom), with drivermlx5_coreorbnxt_en. On a GPU node that matches all 8 ConnectX-7 NICs. rdmaHcaMax: 1000: the node advertises 1000 units (rdma/hca_devices: 1kinkubectl describe node). These are shares, not NICs. Up to 1000 containers can hold one each.- What one unit gives. The kubelet’s device-plugin checkpoint on a node shows the Allocate response for the prefill container:
/dev/infiniband/uverbs0…uverbs7,umad0…umad7and/dev/infiniband/rdma_cm. One unit means access to all 8 NICs, shared with every other Pod on the node.
So the RDMA side is not isolated per Pod or per GPU. Which NIC a process actually uses is chosen by the application (the transfer library’s device configuration), not by Kubernetes.
The RDMA plugin DaemonSet also mounts a containerd drop-in, /etc/systemd/system/containerd.service.d/rdma-memlock.conf with LimitMEMLOCK=infinity. RDMA registers (pins) memory with the NIC, and a low default locked-memory limit makes large registrations fail. Containers inherit the limit from containerd.
Which address RDMA uses
RoCE v2 addresses peers by GID (a 128-bit identifier that carries an IP address). The GID table of rdma4 lists only entries for ens5np0 itself: RoCE v1 and v2 entries for its link-local address and for its host IP. The Pod’s net1 address is not in it. So net1 gives the Pod an IP on the data network; RDMA connections are made with the NIC’s own GIDs. Which GID index the application picks is its own configuration (not verified here).
Failure modes
| Symptom | Likely cause | Check |
|---|---|---|
FailedCreatePodSandBox, “cannot find a network-attachment-definition” |
NAD missing in this cluster (a Deployment names a NAD that only exists in another cluster) | kubectl get net-attach-def -A |
| Sandbox fails, macvlan error on master | per-node file names a NIC that is down or absent | cat /etc/cni/multus/net.d/<nad>.conf; ip link show ensNnp0 |
| Sandbox fails in IPAM | whereabouts range full or its API calls fail | kubectl -n kube-system get ippools.whereabouts.cni.cncf.io |
Pod Pending, Insufficient rdma/hca_devices |
RDMA plugin not registered on the node (allocatable 0) | kubectl get node <n> -o jsonpath='{.status.allocatable}' |
| RDMA calls fail with memory registration errors | memlock limit not applied (containerd not restarted after the drop-in) | systemctl show containerd -p LimitMEMLOCK |
| Node tainted not-ready by a readiness controller | a node health check found Multus or RDMA unhealthy | kubectl describe node <n> conditions and taints |
NVIDIA Network Operator for comparison
NVIDIA packages the same parts for ConnectX and BlueField NICs as the Network Operator. One cluster-scoped NicClusterPolicy (it must be named nic-cluster-policy) with optional sub-states:
| Sub-state | Sample-cluster equivalent |
|---|---|
ofedDriver: DOCA/MLNX-OFED driver container |
in-box distribution mlx5 driver and rdma-core host packages |
rdmaSharedDevicePlugin |
the same plugin, deployed as a plain DaemonSet |
sriovDevicePlugin: one VF per Pod |
not used |
ibKubernetes: InfiniBand partition keys |
not used (Ethernet) |
secondaryNetwork: Multus, CNI plugins, IPoIB CNI |
Multus + standard CNI plugins |
nvIpam: NVIDIA IPAM |
whereabouts |
nicConfigurationOperator: NIC firmware settings |
not managed from Kubernetes |
Extra CRDs (MacvlanNetwork, HostDeviceNetwork, IPoIBNetwork) generate NADs for you. For GPU memory over RDMA (GPUDirect RDMA), the NVIDIA GPU Operator can build the nvidia-peermem module (driver.rdma.enabled) or rely on DMA-BUF. Inside an NVL72 rack, GPUs talk over NVLink, so the NIC path is only for traffic leaving the rack. See rack-scale GPUs.
Look for yourself
kubectl get network-attachment-definitions -A
kubectl -n gpu-demo get pod <prefill> -o jsonpath='{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/network-status}'
kubectl -n kube-system get cm rdma-devices -o jsonpath='{.data.config\.json}'
kubectl -n kube-system get ippools.whereabouts.cni.cncf.io
# on a GPU node (read-only)
cat /etc/cni/net.d/00-multus.conf; ls /etc/cni/multus/net.d
rdma link; ibv_devices
cat /sys/class/net/ens5np0/device/numa_node