Learn · GPU fundamentals
GPU partitioning: AMD modes and NVIDIA MIG
An MI355X has 256 compute units and 288 GB of memory. An 8-billion-parameter model in BF16 needs about 16 GB. Giving it a whole GPU wastes most of the card. Partitioning splits one physical GPU into several logical GPUs, each with its own compute and memory, so several small jobs share a card with hardware isolation.
Large-model serving usually uses whole GPUs: every MI355X runs in SPX/NPS1 and Kubernetes sees 8 devices per node. Partitioning still matters to operators. It decides how many devices the node advertises, what a “GPU” in a claim, a metric or an alert actually is, and changing it is as disruptive as a driver change.
The idea first
Think of one MI355X as an office floor with 8 rooms (the XCDs) and two storerooms (the HBM on each of the two IODs). Partitioning is how you let the floor:
| Setting | Office picture | The question it answers |
|---|---|---|
| Compute partition (SPX, DPX, QPX, CPX) | one company rents all 8 rooms, or 2, 4 or 8 companies rent equal blocks of rooms | “How many GPUs does software see, and how big is each?” |
| Memory partition (NPS1, NPS2) | everyone’s files are spread over both storerooms, or each half of the floor uses the storeroom next to it | “Where does each logical GPU’s memory physically come from?” |
Where the picture breaks: the tenants still share the building’s power, front door and fire alarm (one PSP, one power budget, one XGMI hive), so a fault that resets the card resets every tenant. And the memory setting does not change how much memory each tenant gets, only where it lives: each logical GPU gets an equal share of the card’s 288 GB either way.
Try every combination. The colours show which XCDs form one logical GPU; the yellow bars are the memory pools.
The counts follow the example amd-smi partition output below. Memory per logical GPU is the card’s HBM divided equally, the rule AMD documents for MI300X (CPX: 8 × 24 GB of 192 GB); the MI355X split is assumed to follow the same rule. Which XCD sits on which IOD is drawn schematically.
AMD: two independent settings
On MI300-family and MI350-family GPUs, two settings decide how the chiplets are exposed. They follow the physical layout in what a GPU is: compute partitions follow XCD lines, memory partitions follow IOD lines.
Compute (accelerator) partition
SPX (single) is 1 logical GPU of 8 XCDs and 256 CUs, the default, for big models; DPX (dual) 2 × 128 CUs and 144 GB; QPX (quad) 4 × 64 CUs and 72 GB; CPX (core) 8 × 32 CUs and 36 GB, for many tiny models. amd-smi partition lists the profiles the hardware offers, with the resources each logical GPU gets. Illustrative output, trimmed to GPU 0 (* marks the current profile):
ACCELERATOR_PARTITION_PROFILES:
GPU_ID PROFILE_INDEX MEMORY_PARTITION_CAPS ACCELERATOR_TYPE NUM_PARTITIONS RESOURCE_TYPE RESOURCE_INSTANCES
0 0 NPS1 SPX* 1 XCC 8
DMA 16
1 NPS1,NPS2 DPX 2 XCC 4
DMA 8
2 NPS1,NPS2 QPX 4 XCC 2
DMA 4
3 NPS1,NPS2 CPX 8 XCC 1
DMA 2
An XCC (accelerated compute core) is the schedulable compute unit of one XCD; on these GPUs there is one XCC per XCD. The logical GPU a partition produces is an XCP. In SPX, the command processor spreads a kernel’s workgroups round-robin over all 8 XCCs; in CPX, each XCP runs its kernels on its own XCC only, which keeps data in that XCD’s L2 cache.
Memory partition (NPS)
NPS stands for NUMA nodes per socket (“socket” here means the GPU package). NPS1 is one 288 GB pool, addresses interleaved over all 8 HBM stacks on both IODs: simplest, but every access may cross the IOD-to-IOD link. NPS2 is two 144 GB pools, one per IOD; the XCDs on that IOD use their local pool, so there is less cross-IOD traffic, lower latency and lower power. AMD’s white paper calls DPX + NPS2 the most efficient MI350 mode. MI300X had NPS4 (four IODs); MI355X has two IODs, so it stops at NPS2.
There cannot be more memory pools than compute partitions, so SPX + NPS2 is not offered (the output above lists only NPS1 for SPX); the other seven combinations are. The ROCm compatibility matrix lists the configurations AMD supports in bare-metal deployments; check it before relying on one.
Worked example: one 8-GPU MI355X node in CPX + NPS2
What would an 8-GPU node look like cut into the smallest pieces? Every number is computed from the card’s counts (memory per partition as explained above; the Kubernetes and render-node shapes are expected, not observed).
From 8 whole GPUs to 64 logical ones
1 · Today: 8 cards in SPX + NPS1
- Logical GPUs
- 8
- 1 per card
- Each
- 256 CUs
- 8 XCDs, 16 DMA engines
- Each
- 288 GB
- 294,912 MiB
- Render nodes used
- 8 of 64
- pre-created by the driver
The command processor spreads each kernel’s workgroups round-robin over all 8 XCCs (an XCC, accelerated compute core, is the schedulable compute part of one XCD). Kubernetes sees 8 devices.
2 · Compute: each XCD becomes its own GPU
| XCD a | XCD b | XCD c | XCD d | |
|---|---|---|---|---|
| IOD 0 | 0 | 1 | 2 | 3 |
| IOD 1 | 4 | 5 | 6 | 7 |
In CPX the card becomes 8 XCPs (the logical GPU a partition produces), each with 1 XCC, 32 CUs and 2 DMA engines. A kernel launched on an XCP runs on its one XCD only, which keeps its data in that XCD’s 4 MB L2 cache.
3 · Memory: two 144 GB pools, 4 XCPs each
| XCP | XCP | XCP | XCP | |
|---|---|---|---|---|
| pool 0 (IOD 0) | 36 | 36 | 36 | 36 |
| pool 1 (IOD 1) | 36 | 36 | 36 | 36 |
144 GB ÷ 4 = 36 GB each (36,864 MiB).
NPS2 splits the 8 HBM stacks into one pool per IOD. The 4 XCPs on IOD 0 take their memory only from its 4 stacks, so their traffic never crosses to the other IOD. In NPS1 each XCP would still get 36 GB, interleaved over all 8 stacks.
4 · The node: 64 logical GPUs
| XCP 0 | XCP 1 | XCP 2 | XCP 3 | XCP 4 | XCP 5 | XCP 6 | XCP 7 | |
|---|---|---|---|---|---|---|---|---|
| card 0 | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
| card 1 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 |
| card 2 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 |
| card 3 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 |
| card 4 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 |
| card 5 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 |
| card 6 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 |
| card 7 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 |
8 cards × 8 = 64 entries in amd-smi list and HIP, IDs 0 to 63 (AMD documents the same for 8 × MI300X in CPX). Logical GPUs of one card share its PCI bus and device; the function digit tells them apart (for example 0000:2f:00.0 to 0000:2f:00.7 for card 1, following AMD’s MI300X example). “GPU 5” now means card 0, XCP 5, not the sixth card: every index in metrics, HIP_VISIBLE_DEVICES and logs changes meaning.
5 · Device files: all 64 render nodes in use
- Render nodes backed
- 64 of 64
- Card 0 (example)
- renderD128 + 136–142
- primary + 7 spares today
- /dev/kfd
- 1
- still one for the node
The driver already created these nodes at boot, even in SPX. In this example the 8 primaries are renderD128 to renderD135 and the 56 spares (platform-amdgpu_xcp_*) follow from renderD136; card 0’s spares are renderD136 to renderD142. Real hosts can number them differently, so map through /dev/dri/by-path. In CPX each of the 8 per card backs a logical GPU. A container must get the right one through CDI.
6 · Kubernetes: 64 smaller devices
- ResourceSlice devices
- 8 → 64
- computeUnits
- 256 → 32
- memory
- 36,864Mi
- was 294,912Mi; expected
- partitionProfile
- cpx_nps2
- expected value
The DRA driver would republish the node’s ResourceSlice with 64 devices, and every claim that selects on computeUnits or memory would match differently. The node labeller’s labels (amd.com/gpu.cu-count, beta.amd.com/gpu.vram.288G=8) change too. The exact shape of a partitioned ResourceSlice is not verified here.
7 · What fits now
- 8B model, BF16
- fits
- 16 GB + 20 GB for KV cache
- 70B model, FP8
- does not fit
- 70 GB > 36 GB
- Copies of an 8B model
- 64 per node
- instead of 8
A 70B model would need tensor parallelism across at least two partitions, which defeats the purpose. That arithmetic is why partitioning suits small models and many tenants, and why large models use whole GPUs.
Counts from the example amd-smi partition output; per-partition memory from AMD's documented equal split; device names and ResourceSlice shape after repartitioning are illustrative.
What changes on the node
Device files and enumeration. As in the worked example: the 64 pre-created render nodes (one primary plus 7 spare per GPU; 56 amdgpu_xcp_* platform devices) start backing logical GPUs, /dev/kfd stays one file, and amd-smi list and HIP list one entry per XCP, so “GPU 5” in SPX is not “GPU 5” in CPX.
Metrics. AMD SMI treats XCP 0 as the primary partition: only it reports device-wide values such as board power. Other partitions report only their own utilization and clocks. Socket-level numbers (power, temperature) describe the whole card even when read through one partition.
Kubernetes. The AMD DRA driver publishes each device with a partitionProfile attribute. On an unpartitioned MI355X every device reads:
partitionProfile: {string: spx_nps1}
capacity:
computeUnits: {value: "256"}
memory: {value: 294912Mi}
Fault domains do not shrink. Partitions share one package, one PSP, one power budget and one XGMI hive. A fault that resets the card resets every partition on it, and a hive reset still takes all 8 cards.
How a change is applied, and why it is disruptive
From AMD’s partitioning documentation:
- The GPU must be idle: no process on any partition of the physical GPU.
- Compute partition:
amd-smi set --compute-partition <mode>(root). It takes effect without a reboot. AMD’s MI300X guide notes the GPU returns to SPX after a reboot or driver reload unless something sets it again. - Memory partition:
amd-smi set --memory-partition <mode>, then a driver reload (modprobe -r amdgpu && modprobe amdgpu;amd-smi reset -ris not available in newer ROCm releases). The reload reconfigures every GPU in the hive at once. - Re-enumerate: all old device handles and indexes are invalid.
In Kubernetes, the AMD GPU Operator’s Device Config Manager (DCM) automates this per node. Its documented flow, step by step (it runs only when configManager.enable is true; with it off, nothing can repartition a GPU in use):
- Before8 inference Pods run on 8 whole GPUs; the ResourceSlice lists 8 devices.
- 1 · Taint: evict everythingThe operator taints the node
amd-dcm=up:NoExecute. Every GPU workload is evicted: the GPUs must be idle. - 2 · Label the profileThe operator sets
dcm.amd.com/gpu-config-profile=<profile>. DCM sees the label and starts. - 3 · ApplyDCM sets the compute and memory modes; a memory-mode change reloads the driver for the whole hive. The DRA driver republishes the ResourceSlice with the new devices. A memory-mode change also reloads the driver.
- 4 · UntaintThe operator removes the taint. Pods schedule again, onto the new, smaller devices. A partition change is therefore a node maintenance operation, the same class as a driver upgrade.
The read-only sysfs view, one line per GPU (example output for GPU 0):
0000:11:00.0 compute=SPX mem=NPS1 avail_c=SPX, DPX, QPX, CPX avail_m=NPS1, NPS2
NVIDIA: MIG, MPS and time-slicing
MIG (Multi-Instance GPU)
MIG cuts a GPU into hardware-isolated GPU instances. The building blocks are slices: a memory slice is about 1/8 of the GPU’s memory capacity and bandwidth (with its memory controllers and cache), an SM slice about 1/7 of the SMs. A GPU instance combines memory and SM slices plus copy and media engines; it can be split further into compute instances that share the instance’s memory. Profiles are named <compute slices>g.<memory>gb.
Supported GPU instance profiles on B300 (288 GB), from NVIDIA’s MIG guide:
| Profile | Max instances | Memory | SMs |
|---|---|---|---|
1g.34gb |
7 | 1/8 | 1/7 |
1g.34gb+me (with media engines) |
1 | 1/8 | 1/7 |
1g.67gb |
4 | 2/8 | 1/7 |
2g.67gb |
3 | 2/8 | 2/7 |
3g.135gb |
2 | 4/8 | 3/7 |
4g.135gb |
1 | 4/8 | 4/7 |
7g.269gb |
1 | full | full |
The GB300 (252 GB) product listed in the guide, the GPU in DGX Station, has the same shape with 1g.31gb up to 7g.251gb. MIG profiles for the GB300 in NVL72 were not checked for this page.
Operational facts from the guide:
- Since Hopper, enabling MIG mode does not need a GPU reset. MIG mode is not persistent across reboots: it lasts while the driver is loaded, so something (the GPU Operator’s MIG manager, a systemd unit) must set it at boot.
- MIG devices are reached through extra device nodes under
/dev/nvidia-caps/, which must also go into the container. - Peer-to-peer between MIG instances on one GPU works from driver R570.
- Creating or destroying instances needs the affected instances idle;
nvidia-smi migcommands do it.
MPS and time-slicing
| Method | What is shared | Isolation |
|---|---|---|
| MIG | nothing: own SMs, memory, cache, bandwidth | hardware; a fault in one instance does not affect the others (except whole-GPU faults) |
| MPS (Multi-Process Service) | the GPU, concurrently, through one server process | separate address spaces, optional limits on SM share; memory bandwidth contended |
| Time-slicing | the GPU, one process at a time in turns | none for memory capacity; the Kubernetes device plugin simply advertises one GPU several times |
AMD and NVIDIA side by side
| AMD MI355X | NVIDIA B300 / GB300 | |
|---|---|---|
| Unit of split | XCD (8 per card) | GPU slice (7 SM slices, 8 memory slices) |
| Max logical GPUs | 8 (CPX) | 7 (1g profiles) |
| Memory split | NPS1 or NPS2, by IOD | per instance, in eighths |
| Mixed sizes on one card | no; all partitions in one mode are equal | yes, any valid placement of profiles |
| Change compute layout | idle GPU, no reboot | idle instances, no reset on Hopper+ |
| Change memory layout | driver reload, whole hive | part of the profile |
| Persistence | compute resets to SPX on driver reload (per MI300X guide) | MIG mode lost on reboot |
| Kubernetes automation | GPU Operator Device Config Manager | GPU Operator MIG manager |
Why this matters for operations
- Device count is a function of mode. A node is “8 GPUs” only in SPX. Capacity planning, dashboards and claim templates must state the mode.
- Names move. Index, render node and amd-smi ID change with the mode. Use PCI address plus partition ID when you correlate.
- Repartitioning is maintenance. Drain, change, verify the ResourceSlice, uncordon. On AMD a memory-mode change reloads the driver for the whole hive.
- Isolation is partial. Partitions protect against noisy neighbours in compute and memory capacity, not against card-level or hive-level faults.
Further reading
- AMD CDNA 4 white paper: compute and memory partitioning (pages 10–12)
- AMD SMI: GPU partitioning concepts · MI300X partitioning overview · quick start
- ROCm compatibility matrix: partitioning support
- AMD GPU Operator: applying partition profiles (DCM)
- NVIDIA MIG user guide: concepts · supported profiles · getting started