How to read. The map runs from the smallest building blocks (bottom: threads, PTX, tokens, attention) up to overviews (top). Each page shows what it is part of above and what it is built from below. New to GPUs? Start at What a GPU is or Tokens and transformers. Short on time? Read the overviews (green): GPU fundamentals, LLM inference and GPUs in Kubernetes.
Pages cover AMD and NVIDIA side by side.
Topic map
Kernels & silicon7 pages
Inside a computer, level by level
A desktop computer as one diagram you can open up: CPU down to a single transistor, RAM down to one capacitor, the SSD down to one flash cell, plus the GPU, chipset, keyboard, microphone and power supply.
Read more: Inside a computer, level by levelHow a GPU runs code: threads, warps, wavefronts
The GPU execution model from the bottom: a kernel launch is a grid of thread blocks; blocks run on one CU/SM; threads run in lockstep groups (warp 32, wavefront 64) under an execution mask; registers and LDS decide how many groups fit (occupancy), and those groups hide memory latency. Worked occupancy math for MI355X and H100.
Read more: How a GPU runs code: threads, warps, wavefrontsFrom source to silicon: PTX, SASS and AMDGCN
What the GPU actually executes and how it gets there. NVIDIA compiles CUDA to PTX (a virtual ISA) and then SASS per architecture, packed in a fatbinary with JIT fallback; AMD compiles HIP through LLVM to AMDGCN code objects per gfx target (gfx950 for MI355X) in an offload bundle. Real compiler output for loads, LDS, matrix-core ops (mma, wgmma, tcgen05, v_mfma), and how to dump and disassemble them yourself.
Builds on: Threads & wavefronts
Read more: From source to silicon: PTX, SASS and AMDGCNGPU memory hierarchy and the roofline
Registers, LDS/shared memory, L1, L2, Infinity Cache and HBM with sizes per level for MI355X, H100 and B200; coalescing and bank conflicts; and the roofline: arithmetic intensity (FLOPs per byte) against ridge points, worked for real kernels and for LLM prefill and decode.
Builds on: Threads & wavefronts
Read more: GPU memory hierarchy and the rooflineGEMM tiling: how a matrix multiply fills the matrix cores
Why a naive matrix multiply is memory-bound and how tiling fixes it: workgroup tiles in LDS, wave tiles in registers, MFMA/Tensor Core instructions, double buffering, the epilogue, tile-count problems on skinny decode shapes (split-K), and grouped GEMM for mixture-of-experts. Worked numbers for MI355X.
Builds on: PTX & AMDGCN ISA, Memory & roofline
Read more: GEMM tiling: how a matrix multiply fills the matrix coresFlashAttention and paged attention: attention as GPU kernels
Why naive attention is memory-bound, how FlashAttention computes exact attention tile by tile with the online softmax (worked numeric example), how the kernel uses LDS and matrix cores, and how decode-time paged attention differs: one query per sequence, KV scattered in blocks, GQA reuse, split-KV for long contexts.
Builds on: GEMM tiling, Attention & KV cache
Read more: FlashAttention and paged attention: attention as GPU kernelsPrefill and decode as GPU kernels
One gpt-oss-120b transformer layer as the kernels a GPU actually runs (norms, GEMMs, RoPE, KV-cache write, attention, MoE routing and grouped GEMMs), their shapes in prefill vs decode, what changes at kernel level between the two, real compiled PTX and AMDGCN for the KV-cache write, and how to see real kernels with rocprofv3 and Nsight.
Builds on: GEMM tiling, FlashAttention & paged attn, Prefill, Decode
Read more: Prefill and decode as GPU kernels
GPU fundamentals8 pages
What a GPU is
The physical picture of a datacenter GPU for a newcomer: chiplets (AMD XCD/IOD, NVIDIA dual die), CUs and SMs, matrix engines, HBM stacks, power and heat, host versus device. AMD MI355X and NVIDIA Blackwell Ultra (GB300) side by side, with illustrative node output.
Builds on: Threads & wavefronts, Memory & roofline, Number formats
Read more: What a GPU isGPU number formats: FP16 to FP4 and MX
How a GPU stores a number in 16, 8, 6 or 4 bits, why fewer bits mean more speed and a smaller model, and what block scaling (OCP MXFP, NVIDIA NVFP4) adds. With an interactive encoder, one weight encoded bit by bit, one 32-value MXFP4 block worked end to end, and MI355X and Blackwell Ultra numbers.
Read more: GPU number formats: FP16 to FP4 and MXGPU boot and firmware: from power-on to a kernel launch
What happens between pressing power and a kernel running on a datacenter GPU: on-board firmware (AMD IFWI, PSP, SMU; NVIDIA VBIOS, GSP), driver probe, device files, queues and doorbells. AMD MI355X with illustrative node output, NVIDIA side by side, and the failure modes each step owns.
Builds on: What a GPU is
Read more: GPU boot and firmware: from power-on to a kernel launchGPU memory and interconnects
Every link a byte can cross, fastest first: HBM inside the GPU, GPU-to-GPU scale-up links (AMD XGMI hive, NVIDIA NVLink and NVSwitch), PCIe to the host and NIC, and RDMA networks between servers (GPUDirect, peer memory). Speeds, worked transfer times, fault domains, and the MI355X hive in example tool output.
Builds on: What a GPU is
Read more: GPU memory and interconnectsReading GPU node topology: PCIe, NUMA and GPU links
How to find out, on a live node, which CPU cores, NIC and PCIe switch belong to which GPU, and how the GPUs link to each other. Read-only commands for AMD (lspci, sysfs, amd-smi topology, KFD) and NVIDIA (nvidia-smi topo -m, fabric state), with the full map of an illustrative 8-GPU MI355X node.
Builds on: Memory & interconnects
Read more: Reading GPU node topology: PCIe, NUMA and GPU linksThe GPU software stack
Firmware, kernel driver (amdgpu/KFD, NVIDIA open modules, DKMS), device files, runtime (ROCr/HIP, libcuda/CUDA runtime), libraries and frameworks. Which layer is on the host and which in the container, how versions must match (ROCm matrix, CUDA minor and forward compatibility), and what an 8-GPU MI355X server typically has installed (example versions).
Builds on: What a GPU is, PTX & AMDGCN ISA, Boot & firmware
Read more: The GPU software stackGPU partitioning: AMD modes and NVIDIA MIG
How one physical GPU becomes several logical ones. AMD compute partitions (SPX, DPX, QPX, CPX) along XCD lines and memory partitions (NPS1, NPS2) along IOD lines on MI355X; NVIDIA MIG profiles, MPS and time-slicing. What changes in /dev, amd-smi and ResourceSlices, how a change is applied, and why large-model serving keeps GPUs whole.
Builds on: Memory & interconnects, Software stack
Read more: GPU partitioning: AMD modes and NVIDIA MIGGPU fundamentals (overview)
overviewThe physical and software picture of a datacenter GPU in one view: what is in the package, how bytes move, how it boots, the software layers and partitioning. AMD MI355X and NVIDIA GB300 side by side, plus the name map you will keep coming back to.
Combines: What a GPU is, Number formats, Boot & firmware, Memory & interconnects, Node topology, Software stack, Partitioning
Read more: GPU fundamentals (overview)
LLM inference10 pages
Inside a model, level by level
One forward pass of gpt-oss-120b as a diagram you can open up: a layer as GPU kernels, attention with the KV cache, FlashAttention, the MoE router, and a GEMM down to one workgroup's K loop.
Read more: Inside a model, level by levelTokens and transformers: what an LLM computes
From the model's side, step by step: text becomes tokens, tokens become vectors, 36 transformer layers (norm, attention with RoPE, mixture-of-experts MLP, residual adds) update them, and the last vector picks the next token. A real tokenizer run, one layer worked through with numbers, a sampling explorer, and weights, parameter counts and FLOPs for gpt-oss-120b.
Read more: Tokens and transformers: what an LLM computesMixture of experts: 117 B parameters, 5 B per token
How a mixture-of-experts layer routes each token to 4 of 128 expert MLPs in gpt-oss-120b, worked through with numbers: router scores, top-4, softmax, weighted sum. Why total and active parameters differ, simulated expert load per batch, and a calculator for what routing does to bytes read and arithmetic intensity.
Builds on: Tokens & transformers
Read more: Mixture of experts: 117 B parameters, 5 B per tokenAttention and the KV cache
How attention works step by step (queries, keys, values, softmax, causal mask, multi-head and grouped-query attention, sliding windows), why servers keep every past token's keys and values, and exact KV cache size formulas with worked numbers for gpt-oss-120b and Llama 3.3 70B.
Builds on: Tokens & transformers
Read more: Attention and the KV cacheKV cache management: pages, prefixes, quantization, offload
How an inference server stores the KV cache: fixed-size blocks and block tables (PagedAttention), prefix caching by chained block hashes, eviction and preemption when memory runs out, FP8 KV quantization and CPU offload. A step-by-step allocator simulation, a prefix-hit calculator, and the vLLM settings and counters to watch.
Builds on: Attention & KV cache
Read more: KV cache management: pages, prefixes, quantization, offloadPrefill: reading the prompt
Prefill runs every prompt token through the model in one pass, fills the KV cache and sets time to first token. One prefill worked through with numbers, the FLOP formula with a calculator, why it is compute-bound, chunked prefill of a 30,000-token prompt, and head-of-line blocking.
Builds on: Attention & KV cache, KV cache management, Mixture of experts
Read more: Prefill: reading the promptDecode: generating tokens
Decode makes one token per sequence per step, re-reading weights and the whole KV cache each time. One user's decode stepped token by token with the bytes read per step, why decode is bandwidth-bound, a batch-size calculator for gpt-oss-120b and Llama 3.3 70B on an MI355X, and the levers that make decode faster.
Builds on: Attention & KV cache, KV cache management, Mixture of experts
Read more: Decode: generating tokensBatching and scheduling: who runs in the next step
How an inference server decides, every step, which requests run and how many tokens each gets: static versus continuous batching, the token budget, mixing prefill chunks with decodes, admission and preemption, priorities, and how disaggregation changes the problem. A step-by-step simulation, head-of-line blocking, and budget and Little's law calculators.
Read more: Batching and scheduling: who runs in the next stepSpeculative decoding: draft and target
A small draft model guesses several tokens; the big target model checks them all in one pass and keeps the ones it agrees with, with the same output distribution. A worked round with real numbers (greedy and sampling rules, KV kept and dropped), a speedup calculator, EAGLE-style and other variants, and why a disaggregated server needs a draft prefill as well as a target prefill.
Read more: Speculative decoding: draft and targetLLM inference from the model's side (overview)
overviewTokens and layers, mixture of experts, attention and the KV cache, prefill, decode, batching, speculative decoding, the kernels they become, and why the two phases behave so differently: one gpt-oss-120b request followed end to end with numbers.
Combines: Tokens & transformers, Mixture of experts, Attention & KV cache, KV cache management, Prefill, Decode, Batching & scheduling, Speculative decoding, Prefill & decode kernels
Read more: LLM inference from the model's side (overview)
GPUs in Kubernetes7 pages
How Kubernetes hands out GPUs: device plugins and DRA
The two allocation models side by side: the device plugin API (a count per node, the kubelet picks the device) and Dynamic Resource Allocation (named devices with attributes, the scheduler picks). How each one works step by step, why a cluster should run one model per GPU, and how to read the result on a sample cluster.
Builds on: Software stack
Read more: How Kubernetes hands out GPUs: device plugins and DRADRA objects in detail: classes, slices, claims and the scheduler
Every resource.k8s.io/v1 object with YAML trimmed from a sample cluster: the gpu.amd.com DeviceClass, a ResourceSlice device, a ResourceClaimTemplate and an allocated ResourceClaim. CEL selectors, constraints, how the scheduler allocates, admin access, prioritized lists, device taints, and which feature gates are on in Kubernetes 1.35.
Builds on: GPU allocation in k8s
Read more: DRA objects in detail: classes, slices, claims and the schedulerCDI: how a GPU gets into a container
The Container Device Interface: the spec files a DRA driver writes, how containerd turns a CDI device name into device nodes and cgroup rules, which device files an AMD GPU needs (/dev/kfd, card, renderD) and how device names map to PCI addresses. Example specs from a sample node, NVIDIA for contrast, and the failure modes.
Builds on: GPU allocation in k8s
Read more: CDI: how a GPU gets into a containerDRA on the node: registration, prepare, checkpoints and restarts
What the kubelet and the AMD GPU DRA driver do on a GPU node: plugin registration, the NodePrepareResources and NodeUnprepareResources gRPC calls, the two checkpoint files, what a driver or kubelet restart does to ResourceSlices and Pods, and how the DaemonSet is built. Sample files from one node.
Builds on: DRA objects in detail, CDI device injection
Read more: DRA on the node: registration, prepare, checkpoints and restartsGPU operators: AMD DeviceConfig, NVIDIA ClusterPolicy and GPUCluster
A GPU operator installs the per-node GPU software as DaemonSets from one custom resource. The AMD GPU Operator in depth (every DeviceConfig section, the reconcile loop, upgradePolicy, Helm hooks, wrapping the chart), then the NVIDIA GPU Operator's ClusterPolicy and GPUCluster, and the NVIDIA DRA driver with ComputeDomains.
Builds on: GPU allocation in k8s
Read more: GPU operators: AMD DeviceConfig, NVIDIA ClusterPolicy and GPUClusterGPU Pod networking: Multus, macvlan, whereabouts and RDMA devices
How a GPU Pod gets a second interface on the data network and access to the RDMA NICs that carry bulk data such as a KV cache: Multus (thin mode), file-backed NetworkAttachmentDefinitions, macvlan, whereabouts IPAM and the RDMA shared device plugin, with illustrative annotations and config from an example cluster. NVIDIA Network Operator for comparison.
Builds on: Memory & interconnects
Read more: GPU Pod networking: Multus, macvlan, whereabouts and RDMA devicesGPUs in Kubernetes (overview)
overviewFrom a physical GPU to a running Pod: who installs what, how DRA allocates a specific GPU, how the node turns that into device files through CDI, how the Pod gets its RDMA network, and what it all looks like on one GPU node. The map of the track, with a symptom-to-layer table.
Combines: GPU allocation in k8s, DRA objects in detail, CDI device injection, DRA on the node, GPU operators, GPU Pod networking
Read more: GPUs in Kubernetes (overview)
Operating GPU nodes6 pages
AMD GPU tools: amd-smi, rocm-smi, sysfs, RVS, AGFHC
A field guide to looking at AMD Instinct GPUs on a GPU node: a step-by-step reading of example amd-smi output for one MI355X GPU, every useful subcommand, rocm-smi, rocminfo, the sysfs files behind them, kernel-log patterns, and the disruptive validation tools (RVS, AGFHC, TransferBench).
Builds on: Software stack, Partitioning
Read more: AMD GPU tools: amd-smi, rocm-smi, sysfs, RVS, AGFHCNVIDIA GPU tools: nvidia-smi, DCGM, Xid, Fabric Manager, IMEX
The NVIDIA side of the toolbox for DGX, HGX and NVL72 platforms: an AMD↔NVIDIA command explorer by task, nvidia-smi queries, DCGM health and diagnostics, five Xids triaged step by step, the Fabric Manager start order on NVSwitch systems, IMEX, NVSentinel, and the bug report.
Builds on: Software stack, Partitioning
Read more: NVIDIA GPU tools: nvidia-smi, DCGM, Xid, Fabric Manager, IMEXAMD RAS and ECC: how a hardware error becomes a log line
Correctable, uncorrectable and deferred errors, each traced step by step from the HBM cell to the kernel log, counters, exporter and an optional node taint; the RAS blocks; the bad-page budget worked through with a calculator; CPER, ACA banks and AFIDs; MODE1 hive resets. AMD in depth, NVIDIA alongside.
Builds on: AMD tools
Read more: AMD RAS and ECC: how a hardware error becomes a log lineHow GPUs fail
The GPU failure classes (memory errors, hangs and resets, wedged firmware, falling off the bus, heat and power limits, link errors) shown on a node's eight GPUs, how each shows up on AMD and NVIDIA, a blast-radius calculator, and a silent fatal event read signal by signal.
Builds on: AMD RAS and ECC, NVIDIA tools
Read more: How GPUs failGPU metrics and monitoring
Where GPU metrics come from (AMD device-metrics-exporter, NVIDIA dcgm-exporter): one energy counter followed from the GPU to a PromQL answer with worked numbers, a PromQL explorer with example results, every AMD metric family and label, Pod attribution under DRA, the freshness trap, and a starter alert set.
Builds on: GPU operators, GPU failures
Read more: GPU metrics and monitoringFrom a GPU process to its Pod (and back)
Read-only recipes to join the GPU world (indexes, PIDs, BDFs, render nodes) to the Kubernetes world (Pods, ResourceClaims, DRA devices), walked through step by step with illustrative PIDs, cgroup paths and claims from an example MI355X node: which Pod owns GPU 4, which GPU a Pod holds, which physical card a DRA device is, and a one-screen map script.
Builds on: AMD tools, DRA on the node
Read more: From a GPU process to its Pod (and back)
GPU platforms1 page
Rack-scale GPUs: NVL72 and Helios
When the unit is a 72-GPU rack, not an 8-GPU server: compute trays and switch trays, the NVLink domain, NVSwitch, Fabric Manager and NVLSM, IMEX and the Kubernetes ComputeDomain; Helios with UALink over Ethernet on Tomahawk 6 and AMD's fabric manager. Worked bandwidth numbers, failure domains, and what changes for placement, health and upgrades.
Builds on: Memory & interconnects, GPU allocation in k8s
Read more: Rack-scale GPUs: NVL72 and Helios