Glossary

Terms used across the site. In a page, the first use of each one is underlined; hover it for the definition.

ACA bank
AMD's machine-check error registers (status, address, IP id, syndrome), held by the SMU. The only data that says which on-chip block failed. AMD RAS and ECC
acceptance rate
In speculative decoding, the chance the target model keeps a draft token. Expected tokens per round = (1 − α^(k+1)) / (1 − α). Leviathan et al. · Learn: Speculative decoding
active parameters
The parameters used to compute one token. For MoE models far fewer than total: gpt-oss-120b has 116.8 B total, 5.1 B active. gpt-oss model card · Learn: Mixture of experts
admin access
DRA request flag (adminAccess: true) that allocates devices even if other claims use them, for monitoring. Only allowed in namespaces labelled resource.kubernetes.io/admin-access=true. Beta, on in 1.35. DRA objects
AFID
AMD Field ID: a code derived from a CPER record (amd-smi ras --afid) that AMD support uses to match a field failure to a known cause. AMD RAS and ECC · AFID event list
AGFHC
AMD GPU Field Health Check: AMD’s long-running hardware diagnostic (about 6 hours). Used for burn-in and before RMA.
AGPR
Accumulation VGPR (AccVGPR) on AMD CDNA: a second register bank that usually holds MFMA accumulators, written as a[...] in assembly. CDNA4 ISA · Page
amd-smi
AMD’s GPU command-line tool and library: inventory, metrics, processes, topology, RAS, partitions. Replaces rocm-smi. AMD SMI docs
AMDGCN · also gfx950, gfx942
AMD GPU machine instruction set, compiled per chip target: gfx942 (MI300X), gfx950 (MI355X). ROCm docs
amdgpu
The AMD GPU Linux kernel driver. Datacenter hosts usually install it as the amdgpu-dkms package, built by DKMS for the running kernel.
anycast
One IP address announced from several machines at once; the network delivers each connection to one of them.
AQL packet
Architected Queuing Language packet (HSA standard): the 64-byte record a ROCm program writes into a GPU queue to launch a kernel. It holds the workgroup and grid sizes, LDS and scratch sizes, and pointers to the code, the arguments and a completion signal. HSA hsa.h · Software stack
Argo CD · also Argo
A GitOps controller that keeps a cluster in sync with manifests in Git, applying and pruning objects as Git changes.
arithmetic intensity
FLOPs done per byte read from memory. Prefill: thousands; decode at batch 1: about 2.
ATHUB
Address Translation hub on AMD GPUs. On MI300/MI350 its name labels the fatal-error signal path, which is not proof that ATHUB itself failed. AMD RAS and ECC · GPU failures
attention
The layer step where each token mixes in information from earlier tokens: its query is compared with their keys, and their values are averaged with the resulting softmax weights. Attention Is All You Need · Learn: Attention and the KV cache
Bad page retirement
The AMD driver stops using an HBM page after an uncorrectable error and records it in board EEPROM. MI35x flags the GPU after 128 retired pages. AMD RAS and ECC · AMD tools
Bank conflict
Several lanes accessing different addresses in the same LDS/shared-memory bank at once, which serializes the access. CDNA4 LDS has 64 banks of 4 bytes. Page
BAR
Base address register: the PCIe address window through which the CPU and other devices reach a device's memory. MI355X exposes a 512 GiB BAR, so all of HBM is reachable for GPUDirect-style RDMA. Learn: Memory and interconnects
BDF
Bus:device.function, the PCI address of a device, with the PCI domain in front (e.g. 0000:11:00.0). The one GPU name every tool (lspci, sysfs, amd-smi, DRA pciBusID) agrees on. Node topology
BF16
Bfloat16: a 16-bit float with FP32's 8-bit exponent and a 7-bit mantissa. The usual format for training and for many served weights. Learn: Number formats
BGP
Border Gateway Protocol: how routers exchange which IP prefixes are reachable through whom. kube-vip uses it to announce a VIP.
block table
Per-sequence map from logical KV blocks to physical blocks in GPU memory, like a page table; the core of PagedAttention. PagedAttention paper · Learn: KV cache management
BPE
Byte-pair encoding: a tokenizer built by repeatedly merging the most frequent adjacent byte pairs in training text, so common words become one token and rare ones several. tiktoken · Learn: Tokens and transformers
CDI
Container Device Interface: a JSON/YAML spec telling the runtime which device files, mounts and env to add to a container. CDI spec
CEL
Common Expression Language. DeviceClasses and ResourceClaims use CEL expressions over device.driver, device.attributes and device.capacity to select devices. An unknown attribute is an error, not false. DRA objects · cel.dev
CephFS
The POSIX file system of the Ceph distributed storage system; many clients can mount and share the same files.
chunked prefill
Splitting a long prompt's prefill across several engine steps, each capped by a token budget (for example 8,192 tokens). Sarathi-Serve · Learn: Prefill
ClusterPolicy
The NVIDIA GPU Operator custom resource that turns driver, toolkit, device plugin, DCGM and others on or off.
Coalescing
The GPU merging the memory requests of a wave's lanes into as few cache lines as possible. Neighbouring lanes reading neighbouring addresses coalesce fully; scattered or strided addresses fetch many lines for few useful bytes. Memory hierarchy
Code object
An AMDGPU ELF file holding compiled kernels for one gfx target, plus metadata (registers, LDS, wavefront size). Several are packed in a .hip_fatbin offload bundle. LLVM AMDGPU · Page
Cold power cycle
Removing power from the server (BMC off, confirm Off, on) so the GPU baseboard and its firmware restart. A BMC "cycle" (ForceRestart) does not remove power.
compute tray
A 1U sled in a rack-scale system with CPUs, 4 GPUs, NICs and its own BMC; one Linux host and expected to be one Kubernetes node. Rack-scale GPUs
ComputeDomain
NVIDIA DRA driver object that forms an IMEX domain around a workload's Pods so GPUs in different NVL72 trays can share memory over NVLink. Lives as long as the workload. GPU operators · NVIDIA docs
containerd
The container runtime that pulls images and runs containers on each node; it can try a list of registry mirrors in order.
continuous batching
Re-forming the batch at every decode step so finished requests leave and new ones join immediately; also called iteration-level scheduling. Learn: Batching and scheduling
cordon
Marking a node unschedulable (spec.unschedulable). Running Pods stay; only a person uncordons.
CPER
Common Platform Error Record: a UEFI-standard error report the driver builds from hardware error registers; read with amd-smi ras --cper. Fatal ATHUB events write none. AMD RAS and ECC
CU · also Compute Unit
Compute Unit: AMD’s compute block inside a GPU; NVIDIA calls it an SM. MI355X has 256.
CUDA
NVIDIA’s GPU programming platform and runtime. libcuda belongs to the driver; the toolkit ships in containers. CUDA docs
CUDA forward compatibility
Running an application built with a newer CUDA major release on an older data-center driver, by adding the cuda-compat-<major>-<minor> package (a newer libcuda) to LD_LIBRARY_PATH. Forward compatibility · Learn: Software stack
DaemonSet
A Kubernetes workload that runs one copy of a Pod on every node (or every matching node).
DCGM
NVIDIA Data Center GPU Manager: health checks, diagnostics (dcgmi diag) and metrics; dcgm-exporter feeds Prometheus. DCGM docs · dcgm-exporter
DCM
Device Config Manager: the AMD GPU Operator component that applies a GPU partition profile to a node, after the node is tainted so all GPU work is evicted. AMD DCM docs · Partitioning
decode
The second phase of LLM inference: generate output one token at a time.
device plugin
The older Kubernetes way to expose devices: a node agent advertises a count (e.g. amd.com/gpu: 8). Kubernetes docs
DeviceClass
DRA object naming a kind of device by a selector. The AMD driver's is gpu.amd.com.
DeviceConfig
The AMD GPU Operator custom resource. It selects nodes and turns each operand on or off.
DeviceTaintRule
Cluster-scoped DRA object that taints selected devices (NoSchedule, NoExecute or None) without the driver. Alpha and off by default in Kubernetes 1.35 (resource.k8s.io/v1alpha3). DRA objects · Kubernetes: device taints
disaggregated inference
Running prefill and decode on separate pools of hardware and moving the KV cache between them, so each phase can be sized and tuned on its own (DistServe, Splitwise). DistServe paper
Divergence
When threads of one warp/wavefront take different branches; the hardware runs each path with some lanes masked off, so the group pays for both. Page
DKMS
Dynamic Kernel Module Support: rebuilds an out-of-tree kernel module (like amdgpu) for each installed kernel.
Doorbell
A memory-mapped GPU register a process writes to tell the command processor "new packets in my queue". One ordinary store, no system call, which is why kernel launches bypass the Linux driver. Software stack · Boot and firmware
Double buffering
Keeping two LDS buffers in a tiled kernel: loads for the next K-step fill one while the matrix cores compute from the other, so math and memory overlap. Also called software pipelining; more buffers = more stages. GEMM tiling
DRA · also Dynamic Resource Allocation
Dynamic Resource Allocation: the Kubernetes API that lets a driver publish devices and the scheduler give a Pod a specific one through a ResourceClaim. Kubernetes docs · Learn: GPU allocation
draft model
In speculative decoding, the small fast model that proposes several tokens for the target model to verify. Leviathan et al. · Learn: Speculative decoding
E8M0
The 8-bit shared scale of the OCP MX formats: an exponent only (bias 127), so the scale is always a power of two. One E8M0 scale serves a block of 32 elements. Number formats
EAGLE
A speculative decoding method whose light draft head runs on the target model's hidden states and proposes a tree of tokens verified in one pass. EAGLE paper · EAGLE-3
ECC
Error-correcting code memory. Correctable errors are fixed silently; uncorrectable ones corrupt data and usually need a reset.
ECMP
Equal-cost multi-path routing: when several routes to a destination are equally good, the router spreads connections across all of them.
embedding
The lookup that turns a token id into a vector of hidden-size numbers (2,880 for gpt-oss-120b); also the table holding those vectors. Learn: Tokens and transformers
Epilogue
The end of a GEMM kernel: converts FP32 accumulators to the output format and fuses cheap element-wise work (bias, activation, residual add) before storing the tile once. GEMM tiling
EXEC mask
AMD's 64-bit per-wavefront register with one bit per lane; vector instructions only write results for lanes whose bit is 1. How branches and divergence are implemented on CDNA. CDNA4 ISA · Page
expert parallelism
Spreading an MoE model's experts over several GPUs; each MoE layer then needs all-to-all exchanges of token vectors. vLLM expert parallel
Fabric Manager
NVIDIA host service that configures NVSwitch routing. Its version must match the GPU driver exactly. NVIDIA guide
Fatbinary
Container embedded in a host ELF with device code for several GPU architectures: .nv_fatbin (cubins + PTX) on NVIDIA, .hip_fatbin (offload bundle) on AMD. nvcc · Bundler · Page
first fit
How the Kubernetes DRA allocator picks a device: it walks pools and ResourceSlices in lexicographic name order, devices in slice order, and takes the first free device that passes all selectors and constraints. No scoring. DRA objects in detail · Kubernetes docs
FlashAttention
An attention kernel that tiles Q·Kᵀ and softmax in on-chip memory instead of writing the full matrix to HBM.
forward pass
One run of the model over a set of input tokens, producing hidden states (and next-token logits) at every position. Prefill is one forward pass; each decode step is another. Learn: Tokens and transformers
FP8
8-bit floating point, in two OCP variants: E4M3 (max 448, used for inference) and E5M2 (max 57344, used for gradients). Doubles matrix throughput and halves memory versus BF16. Learn: Number formats
Freshness canary
An alert that fires when a counter that must always advance (gpu_energy_consumed) stops changing while the scrape still succeeds: the exporter data is stale. GPU metrics
fsync
A system call that blocks until a file’s data is written to the storage device, so a hung mount makes it hang too.
GEMM · also GEMMs
General matrix-matrix multiply: the core operation of prefill. GEMV (matrix × vector) dominates decode.
GPU · also GPUs
Graphics processing unit: a chip with thousands of small cores and its own fast memory, used for graphics, AI training and LLM inference. Wikipedia · Learn: What a GPU is
GPU kernel · also kernels
A function launched on the GPU as thousands of parallel threads (not the Linux kernel). Learn: threads & wavefronts
GPU operator · also GPU Operator
A controller that installs per-node GPU software as DaemonSets from one custom resource (AMD DeviceConfig, NVIDIA ClusterPolicy). AMD GPU Operator · NVIDIA GPU Operator
GPUCluster
NVIDIA GPU Operator custom resource (singleton gpu-cluster) that manages the NVIDIA DRA driver, ComputeDomains, DCGM Exporter and a DRA validator. A cluster has a GPUCluster or a ClusterPolicy, not both. GPU operators · NVIDIA docs
GPUDirect RDMA
NVIDIA's name for a NIC reading and writing GPU memory directly over PCIe, without staging in host RAM. Needs the GPU and NIC under one root complex and a peer-memory path (nvidia-peermem or dma-buf). GPUDirect RDMA · Learn: Memory and interconnects
GQA
Grouped-query attention: several query heads share one key/value head (gpt-oss-120b: 64 query, 8 KV heads), cutting KV cache size by the group factor. GQA paper · Learn: Attention and the KV cache
Grouped GEMM
One kernel launch that runs many differently sized GEMMs, e.g. one per MoE expert, choosing each tile's weight matrix from an expert-id table. vLLM fused_moe · Page
GSP
GPU System Processor: an on-GPU processor on NVIDIA GPUs that runs firmware shipped with the driver and takes over initialization and management work. The open kernel modules require GSP firmware from the same driver release. NVIDIA GSP firmware · Learn: Boot and firmware
HAProxy
An open-source TCP/HTTP load balancer and proxy. It health-checks its backend servers and only sends traffic to the ones that pass.
HBM
High Bandwidth Memory: DRAM stacked next to the GPU die, several TB/s. An MI355X has 288 GB. Wikipedia
HCA
Host Channel Adapter: an RDMA-capable NIC. Pods ask for one as rdma/hca_devices.
head-of-line blocking
Short requests waiting behind a long one that occupies the step budget. Learn: Prefill
Helios
AMD rack-scale system: 18 trays × 4 MI455X = 72 GPUs on a UALink-over-Ethernet scale-up fabric.
Helm
The Kubernetes package manager: it renders a chart (templates plus values) into objects and installs or upgrades them as a release.
hidden size
Width of each token's vector inside the model (2,880 for gpt-oss-120b, 8,192 for Llama 3.3 70B). gpt-oss config
HIP
AMD’s C++ GPU programming API, close to CUDA; code can be ported between them. HIP docs
hive
AMD: a set of GPUs joined by XGMI. Also a fault domain: a reset of one MI35x GPU can reset the whole 8-GPU hive.
IFWI
Integrated Firmware Image: the firmware on an AMD Instinct GPU's flash that the PSP boots from, shipped inside a PLDM bundle and flashed through the server BMC. ROCm compatibility matrix · Learn: Boot and firmware
IMEX
NVIDIA service that lets GPUs in different trays of an NVLink domain share memory; NVIDIA DRA exposes it as a ComputeDomain. IMEX guide
Infinity Cache
AMD's last-level cache shared by all XCDs of an MI300/MI355-class GPU; 256 MB on MI355X; rocminfo reports it as L3. ROCm specs · Page
IOD
I/O die: the base chiplet under the XCDs in AMD Instinct GPUs. It holds the Infinity Cache, HBM controllers, Infinity Fabric links and PCIe. MI355X has two; NPS2 memory partitions follow IOD lines. AMD CDNA 4 white paper
ITL
Inter-token latency: time between two consecutive output tokens of one request, about one decode step. Also called time per output token (TPOT). vLLM metrics · Learn: Decode
Kernarg buffer
Kernel-argument buffer: memory the GPU can read that holds a kernel launch's arguments (buffer addresses, sizes). The AQL packet carries a pointer to it. Software stack
Kernarg segment
AMD: the memory buffer holding a kernel's launch arguments (plus hidden ones such as block size). Its address arrives in s[0:1]; code reads it with s_load. NVIDIA uses constant bank 0 instead. PTX and AMDGCN
Kernel descriptor
AMD: a 64-byte block (symbol <kernel>.kd) in the code object that tells the command processor how to start a kernel: code address, register counts, LDS and argument sizes. PTX and AMDGCN
KFD
Kernel Fusion Driver: the compute part of amdgpu. Its device file /dev/kfd is given to every GPU container.
KMM
Kernel Module Management: an operator that builds and loads kernel modules in containers. An alternative to installing the driver on the host. KMM docs
kube-vip
A Kubernetes component that provides a virtual IP for the cluster, announcing it with ARP or BGP from the nodes that hold it.
kubelet
The agent on each Kubernetes node that starts and watches the Pods scheduled there and runs their probes.
kubelet plugin
A node-local gRPC server a DRA driver runs next to the kubelet. It registers through /var/lib/kubelet/plugins_registry and answers NodePrepareResources and NodeUnprepareResources. DRA on the node · Kubernetes: how DRA works
KV cache
Per-layer keys and values for every token seen so far; prefill builds it and decode reads it each step. PagedAttention paper
label selector
A Kubernetes query over labels (key=value tags); Deployments, Services and many controllers find their objects by selector instead of keeping lists. Labels and selectors
LDS · also shared memory
Local Data Share (AMD) / shared memory (NVIDIA): fast on-chip memory shared by one thread block, used to reuse tiles.
liveness probe · also Liveness
A Kubernetes check of whether a container is still working. When it fails repeatedly, the kubelet kills and restarts the container.
logit
The raw score a model outputs for one vocabulary entry; a softmax over all logits gives next-token probabilities. Learn: Tokens and transformers
macvlan
CNI plugin that creates a child interface with its own MAC on a host NIC and moves it into the Pod. GPU Pod networking · macvlan plugin
Mantissa
The fraction bits of a floating-point number: where the value lies between one power of two and the next. More mantissa bits mean more precision; E4M3 has 3. Number formats
maxSurge
DaemonSet/Deployment rolling-update field: how many extra Pods may exist during an update. 0 means delete the old Pod first, then start the new one; the AMD operator uses 0 for the DRA driver. DaemonSet rolling update · GPU operators
MFMA
Matrix Fused Multiply-Add: AMD matrix-core instructions (v_mfma_*) that multiply small tiles in one step; NVIDIA’s are mma / wgmma.
MFU
Model FLOPs utilization: the share of a chip's peak FLOP/s that a workload actually achieves. Learn: Prefill
MI355X · also MI355, MI350X
AMD Instinct MI355X: CDNA 4 datacenter GPU with 288 GB HBM3E. Usually 8 per server. AMD Instinct docs
MIG · also Multi-Instance GPU
NVIDIA Multi-Instance GPU: splits one GPU into up to 7 hardware-isolated instances. MIG guide
MODE1 reset
The heaviest in-band AMD GPU reset: all blocks reset, VRAM lost, PSP reloads all firmware, power stays on. On MI35x it resets the whole hive. AMD RAS and ECC
MoE · also mixture-of-experts
Mixture of experts: each layer has many expert MLPs and a router picks a few per token; gpt-oss is MoE.
Mooncake
An open-source KV-cache store and transfer engine for LLM serving that pools KV cache across machines and moves it between prefill and decode workers. GitHub
Multus
A CNI plugin that gives a Pod extra network interfaces, each described by a NetworkAttachmentDefinition (NAD). GitHub
MXFP4
OCP microscaling 4-bit format: E2M1 values in blocks of 32 sharing one 8-bit power-of-two (E8M0) scale, about 4.25 bits per value. MXFP6 and MXFP8 use the same scheme. Native on AMD CDNA 4. MX paper · Learn: Number formats
NAD · also NetworkAttachmentDefinition
NetworkAttachmentDefinition: a Multus object describing one extra network.
NCCL
NVIDIA Collective Communications Library: fast data exchange (all-reduce, all-gather) between GPUs. GitHub
NFD · also Node Feature Discovery
Node Feature Discovery: labels nodes with hardware features. GPU operators can use its labels to select GPU nodes. NFD docs
NicClusterPolicy
NVIDIA Network Operator custom resource (name nic-cluster-policy) that deploys OFED driver, RDMA or SR-IOV device plugins, Multus, IPAM and NIC configuration on ConnectX nodes. GPU Pod networking · Network Operator
node labeller
AMD GPU operator operand that writes amd.com/gpu.* and beta.amd.com/gpu.* inventory labels on GPU nodes. Independent of allocation; runs with the device plugin off. GPU operators
NodePrepareResources
DRA gRPC call from the kubelet to a driver before a Pod's containers start. The driver sets up the claim's devices and returns CDI device IDs. Its pair, NodeUnprepareResources, runs after the last Pod using the claim stops. DRA on the node
NPD · also node-problem-detector
node-problem-detector: a node agent that turns health checks and kernel-log patterns into node conditions. GitHub
NPS1 · also NPS2, NPS
AMD memory partition modes (NUMA per socket). NPS1: one pool of HBM per GPU.
NUMA
Non-uniform memory access: a server's CPUs and RAM are split into nodes, and memory or PCIe devices on a far node are slower to reach. Learn: Node topology
NVFP4
NVIDIA's 4-bit float format for Blackwell: E2M1 values with an FP8 (E4M3) scale per 16 values plus an FP32 scale per tensor, about 4.5 bits per value. Inside Blackwell Ultra · Learn: Number formats
nvidia-smi
NVIDIA’s GPU command-line tool: inventory, utilization, processes, ECC, MIG, NVLink and topology. NVIDIA docs
NVL72
NVIDIA rack with 72 Blackwell GPUs and 36 Grace CPUs joined by NVLink switch trays into one domain; arm64. NVIDIA GB300 NVL72
NVLink
NVIDIA’s GPU-to-GPU link. NVSwitch chips join many GPUs into one NVLink domain (72 in an NVL72 rack). NVIDIA
NVLink domain
The set of GPUs that reach each other directly over NVLink: 8 on an HGX baseboard, 72 in a GB300 NVL72 rack. Rack-scale GPUs · NVLink
NVLSM
NVLink Subnet Manager: the service that programs NVSwitch routing tables on Blackwell HGX systems, working alongside Fabric Manager. Fabric Manager guide
NVSwitch
NVIDIA switch chip for NVLink. HGX boards and NVL72 switch trays use it; it needs Fabric Manager on the host.
OAM
OCP Accelerator Module: the mezzanine form factor of MI355X and other datacenter GPUs; eight sit on one universal baseboard (UBB). AMD CDNA 4 white paper
OAM ID
The physical slot of an AMD GPU module on the baseboard, shown by amd-smi list -e. It differs from the GPU index; use it with DC Ops. GPU process ↔ Pod
occupancy
How many warps/wavefronts are resident on a CU/SM, limited by registers and LDS; more hides memory latency.
OIDC
OpenID Connect: token-based login. Kubernetes can trust OIDC tokens from an identity provider so each actor has its own identity. Kubernetes docs
Online softmax
Computing softmax block by block with a running max and sum, rescaling earlier partial results when the max grows; exact, used by FlashAttention. Paper · Page
Operand
One component an operator deploys and keeps in sync, usually as a DaemonSet per enabled section of its custom resource. AMD GPU Operator operands include the DRA driver, node labeller and metrics exporter. GPU operators
PagedAttention · also paged attention
vLLM’s KV-cache scheme: fixed-size blocks plus a per-sequence block table, like virtual memory pages. Paper
PCIe root complex
The CPU's PCIe entry point; every device hangs below one. Devices under the same root (and switch) can exchange data without crossing the CPU. Exposed in DRA as resource.kubernetes.io/pcieRoot. Node topology
PCIe switch
A chip that fans one PCIe link from the CPU out to several devices. In many 8-GPU servers each switch carries one GPU and its NIC, so GPU-to-NIC traffic need not reach the CPU. Learn: Node topology
pcieRoot
Standard DRA device attribute resource.kubernetes.io/pcieRoot: the PCIe root complex a device hangs off. Used in matchAttribute constraints to keep devices together. DRA objects in detail
PDB · also PodDisruptionBudget
PodDisruptionBudget: limits how many Pods of an app voluntary actions (drain) may take down at once. Kubernetes docs
PLDM bundle
The firmware package AMD publishes for Instinct GPUs (IFWI and related images), versioned like 01.00.00.00 and listed per ROCm release in the compatibility matrix. ROCm compatibility matrix
PodResources API
Local kubelet gRPC API (List, Get, GetAllocatableResources) that tells node agents which devices each container got, including DRA devices. Used by GPU metrics exporters to map GPUs to Pods. GPU allocation in k8s
Poison · also deferred error
An uncorrectable error that nobody has read yet. The data is marked poisoned and the system acts only if something consumes it. AMD RAS and ECC
preemption
Evicting a running request's KV cache when memory runs out, to recompute or reload it later; counted by vllm:num_preemptions. vLLM optimization
prefill
The first phase of LLM inference: read the whole prompt in one parallel pass and build the KV cache. Learn: inference basics
prefix caching
Reusing KV blocks of a prompt start that an earlier request already computed, matched by a chained hash of full blocks. vLLM prefix caching · Learn: KV cache management
prioritized list
DRA request form (firstAvailable) that lists alternatives in order; the scheduler takes the first one it can satisfy. Beta, on in Kubernetes 1.35. DRA objects
PSP
Platform Security Processor: the on-GPU security processor on AMD GPUs. It boots from board firmware, verifies and loads microcode for the other blocks, and runs trusted apps such as RAS and XGMI handling. amdgpu driver core · Learn: Boot and firmware
PTX
NVIDIA Parallel Thread Execution: a virtual GPU instruction set; the driver or ptxas compiles it to SASS. CUDA docs (PTX ISA) · Learn page
PVC · also PersistentVolumeClaim
PersistentVolumeClaim: a Kubernetes request for storage that a Pod mounts; it binds to a PersistentVolume.
QKV
Query, key and value: three vectors each token computes per attention head. Queries are compared with keys to weight values. Keys and values are what the KV cache stores. Learn: Attention and the KV cache
RAS
Reliability, availability, serviceability: the hardware error reporting (ECC, fatal events) that amdgpu logs and amd-smi reads.
RBAC
Kubernetes role-based access control: Roles grant verbs on resources; RoleBindings give them to users or service accounts. Kubernetes docs
RCCL
ROCm Communication Collectives Library: AMD’s version of NCCL, for all-reduce and similar across GPUs. GitHub
RDMA
Remote Direct Memory Access: a NIC reads or writes another machine’s memory without its CPU copying the data.
readiness probe · also Readiness
A Kubernetes check of whether a container should receive traffic. When it fails, the Pod is marked NotReady and taken out of Service endpoints, but not restarted.
render node
Per-GPU DRM device file /dev/dri/renderD<N> (minor 128 and up). ROCm can use a GPU only if the process may open its render node, so passing one render node plus /dev/kfd limits a container to that GPU. CDI
reshape_and_cache
vLLM kernel that copies each token's new K and V rows into its paged KV-cache slot given by slot_mapping; one workgroup per token. Source · Page
residual stream
The per-token vectors that flow through all layers; each attention or MLP block adds its output to it instead of replacing it. Learn: Tokens and transformers
ResourceClaim · also ResourceClaims
DRA object: a request for devices. Its status records the allocated device and the Pod using it. Kubernetes docs
resourceclaim controller
Controller in kube-controller-manager that creates one ResourceClaim per Pod from a ResourceClaimTemplate, owns it by the Pod, and deletes it when the Pod is gone. DRA objects
ResourceClaimTemplate
DRA object: a pattern for claims; Kubernetes makes one ResourceClaim per Pod from it.
ResourceSlice · also ResourceSlices
DRA object a driver publishes per node: the devices it has and their attributes.
ResourceSlice wipe
When a DRA plugin stays disconnected longer than the kubelet's 30 s wiping delay (Kubernetes 1.35), the kubelet deletes that node's ResourceSlices for the driver.
ridge point
Peak FLOP/s divided by memory bandwidth: the arithmetic intensity above which a kernel is compute-bound. About 625 FLOP/byte for MI355X at FP8. Roofline model · Learn: Memory hierarchy and roofline
RMSNorm
Normalization that rescales a token's vector to unit root-mean-square, then applies a learned gain; applied before each attention and MLP block in gpt-oss. gpt-oss model card
RoCE
RDMA over Converged Ethernet: RDMA on an Ethernet network. Common for GPU-to-GPU traffic between servers. Wikipedia
ROCm
AMD’s open GPU software platform: HIP runtime, compilers, math and collective libraries, amd-smi. ROCm docs
rocprofv3
AMD's command-line profiler (rocprofiler-sdk): kernel traces with names, durations, VGPR/LDS use, and hardware counters. Docs · Page
roofline
Performance model: a kernel is memory-bound below a FLOPs-per-byte ridge and compute-bound above it. Wikipedia · Learn page
RoPE
Rotary position embedding: rotates query and key vectors by position-dependent angles so their dot product depends on the distance between tokens. RoFormer
router
In an MoE layer, the small matrix that scores all experts for a token and selects the top few (top-4 of 128 in gpt-oss-120b). Learn: Mixture of experts
Row remapping
NVIDIA's graceful HBM wear mechanism: a bad memory row is swapped for a spare. Xid 63 reports a remap, Xid 64 a failed one; read with nvidia-smi -q -d ROW_REMAPPER. AMD's counterpart is bad-page retirement. Xid catalog · NVIDIA tools
running max (m), running sum · also l
The two per-row numbers FlashAttention's online softmax keeps while it streams key blocks: m is the largest score seen so far, l the sum of exp(score − m). Old l and output are rescaled by exp(m_old − m_new) when m grows. FlashAttention worked example
RVS
ROCm Validation Suite: AMD stress and validation tests for GPUs. RVS docs
RWX · also ReadWriteMany
ReadWriteMany: a volume access mode in which many nodes can mount the same volume read-write at once.
SASS
NVIDIA machine code for one GPU architecture (sm_90, sm_100…), produced from PTX.
Saturation
When a value is larger than a format's largest number, it is stored as that largest number instead of infinity. Quantizers use it for formats without infinity (E4M3, FP6, FP4). Number formats
SGPR
Scalar general-purpose register on AMD GPUs: one value shared by the whole wavefront, used for kernel arguments, pointers and loop counters. Run by the scalar unit (s_* instructions). LLVM AMDGPU · Page
sidecar
A helper container that runs next to the main container in the same Pod, sharing its network and volumes.
SIMD unit
A group of 16 lanes in an AMD CU that run one instruction together; a 64-thread wavefront passes through it in 4 cycles. Each MI355X CU has 4. What a GPU is
SIMT
Single Instruction, Multiple Threads: code is written per thread, but hardware fetches one instruction for a whole warp (32) or wavefront (64). Page
sliding-window attention
Attention limited to the most recent N tokens (128 in half of gpt-oss's layers); those layers need KV for only N tokens per sequence. vLLM hybrid KV cache · Learn: Attention and the KV cache
SM · also Streaming Multiprocessor
Streaming Multiprocessor: NVIDIA’s compute block inside a GPU; AMD calls it a CU.
SMU
System Management Unit: the power-management microcontroller on AMD GPUs. It controls clocks, power limits and thermal throttling. amdgpu driver core
speculative decoding
A small draft model guesses tokens; the large target model verifies them in one pass. Paper
Split-K
Splitting a GEMM's K dimension across several workgroups and summing partial results, to use more CUs when M and N are small (decode). Page
Split-KV
Flash-Decoding: splitting one sequence's KV cache across several workgroups in decode attention, then combining partial softmax results. Flash-Decoding · Page
SPX · also CPX, DPX, QPX
AMD compute partition modes. SPX: all compute dies as one GPU. CPX: each die is its own GPU.
StatefulSet
A Kubernetes workload for Pods that need stable names and storage, such as databases or a registry with per-replica state.
SwiGLU
MLP form down(silu(gate(x)) * up(x)) used by gpt-oss and Llama experts and MLPs. gpt-oss model card
switch tray
A rack sled holding scale-up switch chips (NVSwitch on NVL72, Tomahawk 6 on Helios) with its own management CPU and BMC; runs no workloads. Rack-scale GPUs
taint · also taints, NoSchedule
A node mark that repels Pods without a matching toleration. NoSchedule keeps new Pods off; NoExecute also evicts. Kubernetes docs
tcgen05
Blackwell (sm_100a) PTX instruction family for 5th-generation Tensor Cores; tcgen05.mma accumulates into per-SM Tensor Memory. PTX ISA · Page
Tensor Core · also matrix core, Matrix core
Matrix engine inside an SM (NVIDIA) or CU (AMD “matrix core”) that does small matrix multiplies per instruction.
Thanos
A system that extends Prometheus with long-term storage and a global query view; its Ruler evaluates alerting rules.
TMA
Tensor Memory Accelerator: Hopper+ copy engine that moves 1D-5D tiles between global and shared memory from one thread's request (cp.async.bulk.tensor). Hopper tuning · Page
token · also tokens
A word piece from the model’s vocabulary; LLMs read and write text as token IDs.
tokenizer
Splits text into tokens from a fixed vocabulary and maps each to an integer id. gpt-oss uses o200k_harmony, a byte-pair encoding (BPE) with 201,088 entries. HF tokenizer summary · Learn: Tokens and transformers
Tomahawk 6
Broadcom Ethernet switch chip, 102.4 Tb/s with 512 lanes of 200G SerDes. Helios uses 2 per switch tray, 12 per rack. Broadcom
TTFT
Time to first token: mostly prefill time. Its partner metric is ITL, inter-token latency (decode).
UALink
Ultra Accelerator Link: open accelerator-to-accelerator interconnect standard (current spec: UALink 200G 1.0). Helios carries it over Ethernet (UALoE) through Tomahawk 6 switch trays. UALink Consortium
UALoE
UALink over Ethernet: AMD Helios' scale-up fabric, single switch hop through 6 in-rack Tomahawk 6 switch trays. AMD Helios brief
UMC
Unified Memory Controller: the AMD GPU block that drives the HBM stacks and owns HBM ECC. AMD RAS and ECC
VCC
Vector condition code: a 64-bit AMD scalar register pair that a vector compare (v_cmp_*) writes, one bit per lane; bits of lanes that are off are 0. Combined with EXEC to branch. How a GPU runs code
VGPR
Vector general-purpose register on AMD GPUs: one 32-bit value per lane of a wavefront. On MI355X a wave can use up to 512 (256 architectural + 256 accumulation); fewer per wave means more waves fit (occupancy). ROCm specs · Page
VIP
Virtual IP: an address that is not tied to one machine and can move between, or be shared by, several of them.
vLLM
An open-source LLM serving engine. Its engine runs as VLLM::EngineCore processes. vLLM docs
VRAM
The GPU’s own memory (HBM on datacenter GPUs). Tools report how much each process holds.
warp
NVIDIA: 32 threads that execute the same instruction together on one SM.
wavefront
AMD: 64 threads (work-items) that execute the same instruction together on one CU.
Wedge
A GPU state after a failed reset: the devices are still on PCIe but the GPU's boot processor (PSP) never came back, so software cannot use them. On MI35x only removing power (a cold power cycle) recovers it. GPU failures
wgmma
Hopper (sm_90a) PTX instruction: an asynchronous matrix multiply issued by a warpgroup of 128 threads, with operands in shared memory. PTX ISA · Page
whereabouts
Cluster-wide IPAM CNI plugin that stores IP allocations in Kubernetes CRDs (IPPool). A reconciler frees leaked addresses. GPU Pod networking · whereabouts
wiping delay
The 30 s the kubelet waits after a DRA driver disconnects before deleting that node's ResourceSlices for the driver (Kubernetes 1.35 defaultWipingDelay). kubelet DRA manager
XCD
Accelerator complex die: one of the 8 compute chiplets in an AMD MI300/MI350-series GPU. On MI355X each has 36 CUs (32 enabled) and a 4 MB L2. Compute partitions follow XCD lines. AMD CDNA 4 white paper · Learn: What a GPU is
XCP
Accelerated compute processor: the logical GPU produced by an AMD compute partition. SPX gives one XCP per card, CPX gives eight. Each XCP is enumerated as its own GPU by amd-smi and HIP. AMD SMI partitioning · Learn: Partitioning
XGMI · also Infinity Fabric
AMD’s GPU-to-GPU link (Infinity Fabric). The 8 GPUs of an MI355X server form one XGMI hive.
XGMI hive
The set of AMD GPUs joined by one coherent XGMI (Infinity Fabric) network; all 8 GPUs of an MI355X node. A fatal reset covers the whole hive. AMD tools · GPU failures
Xid
An NVIDIA driver error code printed to the kernel log, e.g. Xid 79 (GPU fell off the bus) or 48 (double-bit ECC). Xid catalog