- ACA bank
- AMD's machine-check error registers (status, address, IP id, syndrome), held by the SMU. The only data that says which on-chip block failed. AMD RAS and ECC
- acceptance rate
- In speculative decoding, the chance the target model keeps a draft token. Expected tokens per round = (1 − α^(k+1)) / (1 − α). Leviathan et al. · Learn: Speculative decoding
- active parameters
- The parameters used to compute one token. For MoE models far fewer than total: gpt-oss-120b has 116.8 B total, 5.1 B active. gpt-oss model card · Learn: Mixture of experts
- admin access
- DRA request flag (adminAccess: true) that allocates devices even if other claims use them, for monitoring. Only allowed in namespaces labelled resource.kubernetes.io/admin-access=true. Beta, on in 1.35. DRA objects
- AFID
- AMD Field ID: a code derived from a CPER record (amd-smi ras --afid) that AMD support uses to match a field failure to a known cause. AMD RAS and ECC · AFID event list
- AGFHC
- AMD GPU Field Health Check: AMD’s long-running hardware diagnostic (about 6 hours). Used for burn-in and before RMA.
- AGPR
- Accumulation VGPR (AccVGPR) on AMD CDNA: a second register bank that usually holds MFMA accumulators, written as a[...] in assembly. CDNA4 ISA · Page
- amd-smi
- AMD’s GPU command-line tool and library: inventory, metrics, processes, topology, RAS, partitions. Replaces rocm-smi. AMD SMI docs
- AMDGCN · also gfx950, gfx942
- AMD GPU machine instruction set, compiled per chip target: gfx942 (MI300X), gfx950 (MI355X). ROCm docs
- amdgpu
- The AMD GPU Linux kernel driver. Datacenter hosts usually install it as the amdgpu-dkms package, built by DKMS for the running kernel.
- anycast
- One IP address announced from several machines at once; the network delivers each connection to one of them.
- AQL packet
- Architected Queuing Language packet (HSA standard): the 64-byte record a ROCm program writes into a GPU queue to launch a kernel. It holds the workgroup and grid sizes, LDS and scratch sizes, and pointers to the code, the arguments and a completion signal. HSA hsa.h · Software stack
- Argo CD · also Argo
- A GitOps controller that keeps a cluster in sync with manifests in Git, applying and pruning objects as Git changes.
- arithmetic intensity
- FLOPs done per byte read from memory. Prefill: thousands; decode at batch 1: about 2.
- ATHUB
- Address Translation hub on AMD GPUs. On MI300/MI350 its name labels the fatal-error signal path, which is not proof that ATHUB itself failed. AMD RAS and ECC · GPU failures
- attention
- The layer step where each token mixes in information from earlier tokens: its query is compared with their keys, and their values are averaged with the resulting softmax weights. Attention Is All You Need · Learn: Attention and the KV cache
- Bad page retirement
- The AMD driver stops using an HBM page after an uncorrectable error and records it in board EEPROM. MI35x flags the GPU after 128 retired pages. AMD RAS and ECC · AMD tools
- Bank conflict
- Several lanes accessing different addresses in the same LDS/shared-memory bank at once, which serializes the access. CDNA4 LDS has 64 banks of 4 bytes. Page
- BAR
- Base address register: the PCIe address window through which the CPU and other devices reach a device's memory. MI355X exposes a 512 GiB BAR, so all of HBM is reachable for GPUDirect-style RDMA. Learn: Memory and interconnects
- BDF
- Bus:device.function, the PCI address of a device, with the PCI domain in front (e.g. 0000:11:00.0). The one GPU name every tool (lspci, sysfs, amd-smi, DRA pciBusID) agrees on. Node topology
- BF16
- Bfloat16: a 16-bit float with FP32's 8-bit exponent and a 7-bit mantissa. The usual format for training and for many served weights. Learn: Number formats
- BGP
- Border Gateway Protocol: how routers exchange which IP prefixes are reachable through whom. kube-vip uses it to announce a VIP.
- block table
- Per-sequence map from logical KV blocks to physical blocks in GPU memory, like a page table; the core of PagedAttention. PagedAttention paper · Learn: KV cache management
- BPE
- Byte-pair encoding: a tokenizer built by repeatedly merging the most frequent adjacent byte pairs in training text, so common words become one token and rare ones several. tiktoken · Learn: Tokens and transformers
- CDI
- Container Device Interface: a JSON/YAML spec telling the runtime which device files, mounts and env to add to a container. CDI spec
- CEL
- Common Expression Language. DeviceClasses and ResourceClaims use CEL expressions over device.driver, device.attributes and device.capacity to select devices. An unknown attribute is an error, not false. DRA objects · cel.dev
- CephFS
- The POSIX file system of the Ceph distributed storage system; many clients can mount and share the same files.
- chunked prefill
- Splitting a long prompt's prefill across several engine steps, each capped by a token budget (for example 8,192 tokens). Sarathi-Serve · Learn: Prefill
- ClusterPolicy
- The NVIDIA GPU Operator custom resource that turns driver, toolkit, device plugin, DCGM and others on or off.
- Coalescing
- The GPU merging the memory requests of a wave's lanes into as few cache lines as possible. Neighbouring lanes reading neighbouring addresses coalesce fully; scattered or strided addresses fetch many lines for few useful bytes. Memory hierarchy
- Code object
- An AMDGPU ELF file holding compiled kernels for one gfx target, plus metadata (registers, LDS, wavefront size). Several are packed in a .hip_fatbin offload bundle. LLVM AMDGPU · Page
- Cold power cycle
- Removing power from the server (BMC off, confirm Off, on) so the GPU baseboard and its firmware restart. A BMC "cycle" (ForceRestart) does not remove power.
- compute tray
- A 1U sled in a rack-scale system with CPUs, 4 GPUs, NICs and its own BMC; one Linux host and expected to be one Kubernetes node. Rack-scale GPUs
- ComputeDomain
- NVIDIA DRA driver object that forms an IMEX domain around a workload's Pods so GPUs in different NVL72 trays can share memory over NVLink. Lives as long as the workload. GPU operators · NVIDIA docs
- containerd
- The container runtime that pulls images and runs containers on each node; it can try a list of registry mirrors in order.
- continuous batching
- Re-forming the batch at every decode step so finished requests leave and new ones join immediately; also called iteration-level scheduling. Learn: Batching and scheduling
- cordon
- Marking a node unschedulable (spec.unschedulable). Running Pods stay; only a person uncordons.
- CPER
- Common Platform Error Record: a UEFI-standard error report the driver builds from hardware error registers; read with amd-smi ras --cper. Fatal ATHUB events write none. AMD RAS and ECC
- CU · also Compute Unit
- Compute Unit: AMD’s compute block inside a GPU; NVIDIA calls it an SM. MI355X has 256.
- CUDA
- NVIDIA’s GPU programming platform and runtime. libcuda belongs to the driver; the toolkit ships in containers. CUDA docs
- CUDA forward compatibility
- Running an application built with a newer CUDA major release on an older data-center driver, by adding the cuda-compat-<major>-<minor> package (a newer libcuda) to LD_LIBRARY_PATH. Forward compatibility · Learn: Software stack
- DCGM
- NVIDIA Data Center GPU Manager: health checks, diagnostics (dcgmi diag) and metrics; dcgm-exporter feeds Prometheus. DCGM docs · dcgm-exporter
- DCM
- Device Config Manager: the AMD GPU Operator component that applies a GPU partition profile to a node, after the node is tainted so all GPU work is evicted. AMD DCM docs · Partitioning
- device plugin
- The older Kubernetes way to expose devices: a node agent advertises a count (e.g. amd.com/gpu: 8). Kubernetes docs
- DeviceConfig
- The AMD GPU Operator custom resource. It selects nodes and turns each operand on or off.
- DeviceTaintRule
- Cluster-scoped DRA object that taints selected devices (NoSchedule, NoExecute or None) without the driver. Alpha and off by default in Kubernetes 1.35 (resource.k8s.io/v1alpha3). DRA objects · Kubernetes: device taints
- disaggregated inference
- Running prefill and decode on separate pools of hardware and moving the KV cache between them, so each phase can be sized and tuned on its own (DistServe, Splitwise). DistServe paper
- Divergence
- When threads of one warp/wavefront take different branches; the hardware runs each path with some lanes masked off, so the group pays for both. Page
- DKMS
- Dynamic Kernel Module Support: rebuilds an out-of-tree kernel module (like amdgpu) for each installed kernel.
- Doorbell
- A memory-mapped GPU register a process writes to tell the command processor "new packets in my queue". One ordinary store, no system call, which is why kernel launches bypass the Linux driver. Software stack · Boot and firmware
- Double buffering
- Keeping two LDS buffers in a tiled kernel: loads for the next K-step fill one while the matrix cores compute from the other, so math and memory overlap. Also called software pipelining; more buffers = more stages. GEMM tiling
- DRA · also Dynamic Resource Allocation
- Dynamic Resource Allocation: the Kubernetes API that lets a driver publish devices and the scheduler give a Pod a specific one through a ResourceClaim. Kubernetes docs · Learn: GPU allocation
- draft model
- In speculative decoding, the small fast model that proposes several tokens for the target model to verify. Leviathan et al. · Learn: Speculative decoding
- E8M0
- The 8-bit shared scale of the OCP MX formats: an exponent only (bias 127), so the scale is always a power of two. One E8M0 scale serves a block of 32 elements. Number formats
- EAGLE
- A speculative decoding method whose light draft head runs on the target model's hidden states and proposes a tree of tokens verified in one pass. EAGLE paper · EAGLE-3
- ECC
- Error-correcting code memory. Correctable errors are fixed silently; uncorrectable ones corrupt data and usually need a reset.
- ECMP
- Equal-cost multi-path routing: when several routes to a destination are equally good, the router spreads connections across all of them.
- embedding
- The lookup that turns a token id into a vector of hidden-size numbers (2,880 for gpt-oss-120b); also the table holding those vectors. Learn: Tokens and transformers
- Epilogue
- The end of a GEMM kernel: converts FP32 accumulators to the output format and fuses cheap element-wise work (bias, activation, residual add) before storing the tile once. GEMM tiling
- EXEC mask
- AMD's 64-bit per-wavefront register with one bit per lane; vector instructions only write results for lanes whose bit is 1. How branches and divergence are implemented on CDNA. CDNA4 ISA · Page
- expert parallelism
- Spreading an MoE model's experts over several GPUs; each MoE layer then needs all-to-all exchanges of token vectors. vLLM expert parallel
- Fabric Manager
- NVIDIA host service that configures NVSwitch routing. Its version must match the GPU driver exactly. NVIDIA guide
- Fatbinary
- Container embedded in a host ELF with device code for several GPU architectures: .nv_fatbin (cubins + PTX) on NVIDIA, .hip_fatbin (offload bundle) on AMD. nvcc · Bundler · Page
- first fit
- How the Kubernetes DRA allocator picks a device: it walks pools and ResourceSlices in lexicographic name order, devices in slice order, and takes the first free device that passes all selectors and constraints. No scoring. DRA objects in detail · Kubernetes docs
- FlashAttention
- An attention kernel that tiles Q·Kᵀ and softmax in on-chip memory instead of writing the full matrix to HBM.
- forward pass
- One run of the model over a set of input tokens, producing hidden states (and next-token logits) at every position. Prefill is one forward pass; each decode step is another. Learn: Tokens and transformers
- FP8
- 8-bit floating point, in two OCP variants: E4M3 (max 448, used for inference) and E5M2 (max 57344, used for gradients). Doubles matrix throughput and halves memory versus BF16. Learn: Number formats
- Freshness canary
- An alert that fires when a counter that must always advance (gpu_energy_consumed) stops changing while the scrape still succeeds: the exporter data is stale. GPU metrics
- fsync
- A system call that blocks until a file’s data is written to the storage device, so a hung mount makes it hang too.
- GEMM · also GEMMs
- General matrix-matrix multiply: the core operation of prefill. GEMV (matrix × vector) dominates decode.
- GPU · also GPUs
- Graphics processing unit: a chip with thousands of small cores and its own fast memory, used for graphics, AI training and LLM inference. Wikipedia · Learn: What a GPU is
- GPU kernel · also kernels
- A function launched on the GPU as thousands of parallel threads (not the Linux kernel). Learn: threads & wavefronts
- GPU operator · also GPU Operator
- A controller that installs per-node GPU software as DaemonSets from one custom resource (AMD DeviceConfig, NVIDIA ClusterPolicy). AMD GPU Operator · NVIDIA GPU Operator
- GPUCluster
- NVIDIA GPU Operator custom resource (singleton gpu-cluster) that manages the NVIDIA DRA driver, ComputeDomains, DCGM Exporter and a DRA validator. A cluster has a GPUCluster or a ClusterPolicy, not both. GPU operators · NVIDIA docs
- GPUDirect RDMA
- NVIDIA's name for a NIC reading and writing GPU memory directly over PCIe, without staging in host RAM. Needs the GPU and NIC under one root complex and a peer-memory path (nvidia-peermem or dma-buf). GPUDirect RDMA · Learn: Memory and interconnects
- GQA
- Grouped-query attention: several query heads share one key/value head (gpt-oss-120b: 64 query, 8 KV heads), cutting KV cache size by the group factor. GQA paper · Learn: Attention and the KV cache
- Grouped GEMM
- One kernel launch that runs many differently sized GEMMs, e.g. one per MoE expert, choosing each tile's weight matrix from an expert-id table. vLLM fused_moe · Page
- GSP
- GPU System Processor: an on-GPU processor on NVIDIA GPUs that runs firmware shipped with the driver and takes over initialization and management work. The open kernel modules require GSP firmware from the same driver release. NVIDIA GSP firmware · Learn: Boot and firmware
- HAProxy
- An open-source TCP/HTTP load balancer and proxy. It health-checks its backend servers and only sends traffic to the ones that pass.
- HBM
- High Bandwidth Memory: DRAM stacked next to the GPU die, several TB/s. An MI355X has 288 GB. Wikipedia
- head-of-line blocking
- Short requests waiting behind a long one that occupies the step budget. Learn: Prefill
- Helios
- AMD rack-scale system: 18 trays × 4 MI455X = 72 GPUs on a UALink-over-Ethernet scale-up fabric.
- Helm
- The Kubernetes package manager: it renders a chart (templates plus values) into objects and installs or upgrades them as a release.
- HIP
- AMD’s C++ GPU programming API, close to CUDA; code can be ported between them. HIP docs
- hive
- AMD: a set of GPUs joined by XGMI. Also a fault domain: a reset of one MI35x GPU can reset the whole 8-GPU hive.
- IFWI
- Integrated Firmware Image: the firmware on an AMD Instinct GPU's flash that the PSP boots from, shipped inside a PLDM bundle and flashed through the server BMC. ROCm compatibility matrix · Learn: Boot and firmware
- IMEX
- NVIDIA service that lets GPUs in different trays of an NVLink domain share memory; NVIDIA DRA exposes it as a ComputeDomain. IMEX guide
- Infinity Cache
- AMD's last-level cache shared by all XCDs of an MI300/MI355-class GPU; 256 MB on MI355X; rocminfo reports it as L3. ROCm specs · Page
- IOD
- I/O die: the base chiplet under the XCDs in AMD Instinct GPUs. It holds the Infinity Cache, HBM controllers, Infinity Fabric links and PCIe. MI355X has two; NPS2 memory partitions follow IOD lines. AMD CDNA 4 white paper
- ITL
- Inter-token latency: time between two consecutive output tokens of one request, about one decode step. Also called time per output token (TPOT). vLLM metrics · Learn: Decode
- Kernarg buffer
- Kernel-argument buffer: memory the GPU can read that holds a kernel launch's arguments (buffer addresses, sizes). The AQL packet carries a pointer to it. Software stack
- Kernarg segment
- AMD: the memory buffer holding a kernel's launch arguments (plus hidden ones such as block size). Its address arrives in s[0:1]; code reads it with s_load. NVIDIA uses constant bank 0 instead. PTX and AMDGCN
- Kernel descriptor
- AMD: a 64-byte block (symbol <kernel>.kd) in the code object that tells the command processor how to start a kernel: code address, register counts, LDS and argument sizes. PTX and AMDGCN
- KFD
- Kernel Fusion Driver: the compute part of amdgpu. Its device file /dev/kfd is given to every GPU container.
- KMM
- Kernel Module Management: an operator that builds and loads kernel modules in containers. An alternative to installing the driver on the host. KMM docs
- kube-vip
- A Kubernetes component that provides a virtual IP for the cluster, announcing it with ARP or BGP from the nodes that hold it.
- kubelet
- The agent on each Kubernetes node that starts and watches the Pods scheduled there and runs their probes.
- kubelet plugin
- A node-local gRPC server a DRA driver runs next to the kubelet. It registers through /var/lib/kubelet/plugins_registry and answers NodePrepareResources and NodeUnprepareResources. DRA on the node · Kubernetes: how DRA works
- KV cache
- Per-layer keys and values for every token seen so far; prefill builds it and decode reads it each step. PagedAttention paper
- label selector
- A Kubernetes query over labels (key=value tags); Deployments, Services and many controllers find their objects by selector instead of keeping lists. Labels and selectors
- LDS · also shared memory
- Local Data Share (AMD) / shared memory (NVIDIA): fast on-chip memory shared by one thread block, used to reuse tiles.
- liveness probe · also Liveness
- A Kubernetes check of whether a container is still working. When it fails repeatedly, the kubelet kills and restarts the container.
- logit
- The raw score a model outputs for one vocabulary entry; a softmax over all logits gives next-token probabilities. Learn: Tokens and transformers
- macvlan
- CNI plugin that creates a child interface with its own MAC on a host NIC and moves it into the Pod. GPU Pod networking · macvlan plugin
- Mantissa
- The fraction bits of a floating-point number: where the value lies between one power of two and the next. More mantissa bits mean more precision; E4M3 has 3. Number formats
- maxSurge
- DaemonSet/Deployment rolling-update field: how many extra Pods may exist during an update. 0 means delete the old Pod first, then start the new one; the AMD operator uses 0 for the DRA driver. DaemonSet rolling update · GPU operators
- MFMA
- Matrix Fused Multiply-Add: AMD matrix-core instructions (v_mfma_*) that multiply small tiles in one step; NVIDIA’s are mma / wgmma.
- MFU
- Model FLOPs utilization: the share of a chip's peak FLOP/s that a workload actually achieves. Learn: Prefill
- MI355X · also MI355, MI350X
- AMD Instinct MI355X: CDNA 4 datacenter GPU with 288 GB HBM3E. Usually 8 per server. AMD Instinct docs
- MIG · also Multi-Instance GPU
- NVIDIA Multi-Instance GPU: splits one GPU into up to 7 hardware-isolated instances. MIG guide
- MODE1 reset
- The heaviest in-band AMD GPU reset: all blocks reset, VRAM lost, PSP reloads all firmware, power stays on. On MI35x it resets the whole hive. AMD RAS and ECC
- MoE · also mixture-of-experts
- Mixture of experts: each layer has many expert MLPs and a router picks a few per token; gpt-oss is MoE.
- Mooncake
- An open-source KV-cache store and transfer engine for LLM serving that pools KV cache across machines and moves it between prefill and decode workers. GitHub
- Multus
- A CNI plugin that gives a Pod extra network interfaces, each described by a NetworkAttachmentDefinition (NAD). GitHub
- MXFP4
- OCP microscaling 4-bit format: E2M1 values in blocks of 32 sharing one 8-bit power-of-two (E8M0) scale, about 4.25 bits per value. MXFP6 and MXFP8 use the same scheme. Native on AMD CDNA 4. MX paper · Learn: Number formats
- NAD · also NetworkAttachmentDefinition
- NetworkAttachmentDefinition: a Multus object describing one extra network.
- NCCL
- NVIDIA Collective Communications Library: fast data exchange (all-reduce, all-gather) between GPUs. GitHub
- NFD · also Node Feature Discovery
- Node Feature Discovery: labels nodes with hardware features. GPU operators can use its labels to select GPU nodes. NFD docs
- NicClusterPolicy
- NVIDIA Network Operator custom resource (name nic-cluster-policy) that deploys OFED driver, RDMA or SR-IOV device plugins, Multus, IPAM and NIC configuration on ConnectX nodes. GPU Pod networking · Network Operator
- node labeller
- AMD GPU operator operand that writes amd.com/gpu.* and beta.amd.com/gpu.* inventory labels on GPU nodes. Independent of allocation; runs with the device plugin off. GPU operators
- NodePrepareResources
- DRA gRPC call from the kubelet to a driver before a Pod's containers start. The driver sets up the claim's devices and returns CDI device IDs. Its pair, NodeUnprepareResources, runs after the last Pod using the claim stops. DRA on the node
- NPD · also node-problem-detector
- node-problem-detector: a node agent that turns health checks and kernel-log patterns into node conditions. GitHub
- NUMA
- Non-uniform memory access: a server's CPUs and RAM are split into nodes, and memory or PCIe devices on a far node are slower to reach. Learn: Node topology
- NVFP4
- NVIDIA's 4-bit float format for Blackwell: E2M1 values with an FP8 (E4M3) scale per 16 values plus an FP32 scale per tensor, about 4.5 bits per value. Inside Blackwell Ultra · Learn: Number formats
- nvidia-smi
- NVIDIA’s GPU command-line tool: inventory, utilization, processes, ECC, MIG, NVLink and topology. NVIDIA docs
- NVL72
- NVIDIA rack with 72 Blackwell GPUs and 36 Grace CPUs joined by NVLink switch trays into one domain; arm64. NVIDIA GB300 NVL72
- NVLink
- NVIDIA’s GPU-to-GPU link. NVSwitch chips join many GPUs into one NVLink domain (72 in an NVL72 rack). NVIDIA
- NVLink domain
- The set of GPUs that reach each other directly over NVLink: 8 on an HGX baseboard, 72 in a GB300 NVL72 rack. Rack-scale GPUs · NVLink
- NVLSM
- NVLink Subnet Manager: the service that programs NVSwitch routing tables on Blackwell HGX systems, working alongside Fabric Manager. Fabric Manager guide
- NVSwitch
- NVIDIA switch chip for NVLink. HGX boards and NVL72 switch trays use it; it needs Fabric Manager on the host.
- OAM
- OCP Accelerator Module: the mezzanine form factor of MI355X and other datacenter GPUs; eight sit on one universal baseboard (UBB). AMD CDNA 4 white paper
- OAM ID
- The physical slot of an AMD GPU module on the baseboard, shown by amd-smi list -e. It differs from the GPU index; use it with DC Ops. GPU process ↔ Pod
- occupancy
- How many warps/wavefronts are resident on a CU/SM, limited by registers and LDS; more hides memory latency.
- OIDC
- OpenID Connect: token-based login. Kubernetes can trust OIDC tokens from an identity provider so each actor has its own identity. Kubernetes docs
- Online softmax
- Computing softmax block by block with a running max and sum, rescaling earlier partial results when the max grows; exact, used by FlashAttention. Paper · Page
- Operand
- One component an operator deploys and keeps in sync, usually as a DaemonSet per enabled section of its custom resource. AMD GPU Operator operands include the DRA driver, node labeller and metrics exporter. GPU operators
- PagedAttention · also paged attention
- vLLM’s KV-cache scheme: fixed-size blocks plus a per-sequence block table, like virtual memory pages. Paper
- PCIe root complex
- The CPU's PCIe entry point; every device hangs below one. Devices under the same root (and switch) can exchange data without crossing the CPU. Exposed in DRA as resource.kubernetes.io/pcieRoot. Node topology
- PCIe switch
- A chip that fans one PCIe link from the CPU out to several devices. In many 8-GPU servers each switch carries one GPU and its NIC, so GPU-to-NIC traffic need not reach the CPU. Learn: Node topology
- pcieRoot
- Standard DRA device attribute resource.kubernetes.io/pcieRoot: the PCIe root complex a device hangs off. Used in matchAttribute constraints to keep devices together. DRA objects in detail
- PDB · also PodDisruptionBudget
- PodDisruptionBudget: limits how many Pods of an app voluntary actions (drain) may take down at once. Kubernetes docs
- PLDM bundle
- The firmware package AMD publishes for Instinct GPUs (IFWI and related images), versioned like 01.00.00.00 and listed per ROCm release in the compatibility matrix. ROCm compatibility matrix
- PodResources API
- Local kubelet gRPC API (List, Get, GetAllocatableResources) that tells node agents which devices each container got, including DRA devices. Used by GPU metrics exporters to map GPUs to Pods. GPU allocation in k8s
- Poison · also deferred error
- An uncorrectable error that nobody has read yet. The data is marked poisoned and the system acts only if something consumes it. AMD RAS and ECC
- preemption
- Evicting a running request's KV cache when memory runs out, to recompute or reload it later; counted by vllm:num_preemptions. vLLM optimization
- prefill
- The first phase of LLM inference: read the whole prompt in one parallel pass and build the KV cache. Learn: inference basics
- prefix caching
- Reusing KV blocks of a prompt start that an earlier request already computed, matched by a chained hash of full blocks. vLLM prefix caching · Learn: KV cache management
- prioritized list
- DRA request form (firstAvailable) that lists alternatives in order; the scheduler takes the first one it can satisfy. Beta, on in Kubernetes 1.35. DRA objects
- PSP
- Platform Security Processor: the on-GPU security processor on AMD GPUs. It boots from board firmware, verifies and loads microcode for the other blocks, and runs trusted apps such as RAS and XGMI handling. amdgpu driver core · Learn: Boot and firmware
- PTX
- NVIDIA Parallel Thread Execution: a virtual GPU instruction set; the driver or ptxas compiles it to SASS. CUDA docs (PTX ISA) · Learn page
- PVC · also PersistentVolumeClaim
- PersistentVolumeClaim: a Kubernetes request for storage that a Pod mounts; it binds to a PersistentVolume.
- QKV
- Query, key and value: three vectors each token computes per attention head. Queries are compared with keys to weight values. Keys and values are what the KV cache stores. Learn: Attention and the KV cache
- RAS
- Reliability, availability, serviceability: the hardware error reporting (ECC, fatal events) that amdgpu logs and amd-smi reads.
- RBAC
- Kubernetes role-based access control: Roles grant verbs on resources; RoleBindings give them to users or service accounts. Kubernetes docs
- RCCL
- ROCm Communication Collectives Library: AMD’s version of NCCL, for all-reduce and similar across GPUs. GitHub
- RDMA
- Remote Direct Memory Access: a NIC reads or writes another machine’s memory without its CPU copying the data.
- readiness probe · also Readiness
- A Kubernetes check of whether a container should receive traffic. When it fails, the Pod is marked NotReady and taken out of Service endpoints, but not restarted.
- render node
- Per-GPU DRM device file /dev/dri/renderD<N> (minor 128 and up). ROCm can use a GPU only if the process may open its render node, so passing one render node plus /dev/kfd limits a container to that GPU. CDI
- reshape_and_cache
- vLLM kernel that copies each token's new K and V rows into its paged KV-cache slot given by slot_mapping; one workgroup per token. Source · Page
- residual stream
- The per-token vectors that flow through all layers; each attention or MLP block adds its output to it instead of replacing it. Learn: Tokens and transformers
- ResourceClaim · also ResourceClaims
- DRA object: a request for devices. Its status records the allocated device and the Pod using it. Kubernetes docs
- resourceclaim controller
- Controller in kube-controller-manager that creates one ResourceClaim per Pod from a ResourceClaimTemplate, owns it by the Pod, and deletes it when the Pod is gone. DRA objects
- ResourceClaimTemplate
- DRA object: a pattern for claims; Kubernetes makes one ResourceClaim per Pod from it.
- ResourceSlice · also ResourceSlices
- DRA object a driver publishes per node: the devices it has and their attributes.
- ResourceSlice wipe
- When a DRA plugin stays disconnected longer than the kubelet's 30 s wiping delay (Kubernetes 1.35), the kubelet deletes that node's ResourceSlices for the driver.
- ridge point
- Peak FLOP/s divided by memory bandwidth: the arithmetic intensity above which a kernel is compute-bound. About 625 FLOP/byte for MI355X at FP8. Roofline model · Learn: Memory hierarchy and roofline
- RMSNorm
- Normalization that rescales a token's vector to unit root-mean-square, then applies a learned gain; applied before each attention and MLP block in gpt-oss. gpt-oss model card
- RoCE
- RDMA over Converged Ethernet: RDMA on an Ethernet network. Common for GPU-to-GPU traffic between servers. Wikipedia
- ROCm
- AMD’s open GPU software platform: HIP runtime, compilers, math and collective libraries, amd-smi. ROCm docs
- rocprofv3
- AMD's command-line profiler (rocprofiler-sdk): kernel traces with names, durations, VGPR/LDS use, and hardware counters. Docs · Page
- roofline
- Performance model: a kernel is memory-bound below a FLOPs-per-byte ridge and compute-bound above it. Wikipedia · Learn page
- RoPE
- Rotary position embedding: rotates query and key vectors by position-dependent angles so their dot product depends on the distance between tokens. RoFormer
- router
- In an MoE layer, the small matrix that scores all experts for a token and selects the top few (top-4 of 128 in gpt-oss-120b). Learn: Mixture of experts
- Row remapping
- NVIDIA's graceful HBM wear mechanism: a bad memory row is swapped for a spare. Xid 63 reports a remap, Xid 64 a failed one; read with nvidia-smi -q -d ROW_REMAPPER. AMD's counterpart is bad-page retirement. Xid catalog · NVIDIA tools
- running max (m), running sum · also l
- The two per-row numbers FlashAttention's online softmax keeps while it streams key blocks: m is the largest score seen so far, l the sum of exp(score − m). Old l and output are rescaled by exp(m_old − m_new) when m grows. FlashAttention worked example
- RVS
- ROCm Validation Suite: AMD stress and validation tests for GPUs. RVS docs
- RWX · also ReadWriteMany
- ReadWriteMany: a volume access mode in which many nodes can mount the same volume read-write at once.
- Saturation
- When a value is larger than a format's largest number, it is stored as that largest number instead of infinity. Quantizers use it for formats without infinity (E4M3, FP6, FP4). Number formats
- SGPR
- Scalar general-purpose register on AMD GPUs: one value shared by the whole wavefront, used for kernel arguments, pointers and loop counters. Run by the scalar unit (s_* instructions). LLVM AMDGPU · Page
- sidecar
- A helper container that runs next to the main container in the same Pod, sharing its network and volumes.
- SIMD unit
- A group of 16 lanes in an AMD CU that run one instruction together; a 64-thread wavefront passes through it in 4 cycles. Each MI355X CU has 4. What a GPU is
- SIMT
- Single Instruction, Multiple Threads: code is written per thread, but hardware fetches one instruction for a whole warp (32) or wavefront (64). Page
- sliding-window attention
- Attention limited to the most recent N tokens (128 in half of gpt-oss's layers); those layers need KV for only N tokens per sequence. vLLM hybrid KV cache · Learn: Attention and the KV cache
- SM · also Streaming Multiprocessor
- Streaming Multiprocessor: NVIDIA’s compute block inside a GPU; AMD calls it a CU.
- SMU
- System Management Unit: the power-management microcontroller on AMD GPUs. It controls clocks, power limits and thermal throttling. amdgpu driver core
- speculative decoding
- A small draft model guesses tokens; the large target model verifies them in one pass. Paper
- Split-K
- Splitting a GEMM's K dimension across several workgroups and summing partial results, to use more CUs when M and N are small (decode). Page
- Split-KV
- Flash-Decoding: splitting one sequence's KV cache across several workgroups in decode attention, then combining partial softmax results. Flash-Decoding · Page
- SPX · also CPX, DPX, QPX
- AMD compute partition modes. SPX: all compute dies as one GPU. CPX: each die is its own GPU.
- StatefulSet
- A Kubernetes workload for Pods that need stable names and storage, such as databases or a registry with per-replica state.
- SwiGLU
- MLP form down(silu(gate(x)) * up(x)) used by gpt-oss and Llama experts and MLPs. gpt-oss model card
- switch tray
- A rack sled holding scale-up switch chips (NVSwitch on NVL72, Tomahawk 6 on Helios) with its own management CPU and BMC; runs no workloads. Rack-scale GPUs
- taint · also taints, NoSchedule
- A node mark that repels Pods without a matching toleration. NoSchedule keeps new Pods off; NoExecute also evicts. Kubernetes docs
- tcgen05
- Blackwell (sm_100a) PTX instruction family for 5th-generation Tensor Cores; tcgen05.mma accumulates into per-SM Tensor Memory. PTX ISA · Page
- Tensor Core · also matrix core, Matrix core
- Matrix engine inside an SM (NVIDIA) or CU (AMD “matrix core”) that does small matrix multiplies per instruction.
- Thanos
- A system that extends Prometheus with long-term storage and a global query view; its Ruler evaluates alerting rules.
- TMA
- Tensor Memory Accelerator: Hopper+ copy engine that moves 1D-5D tiles between global and shared memory from one thread's request (cp.async.bulk.tensor). Hopper tuning · Page
- tokenizer
- Splits text into tokens from a fixed vocabulary and maps each to an integer id. gpt-oss uses o200k_harmony, a byte-pair encoding (BPE) with 201,088 entries. HF tokenizer summary · Learn: Tokens and transformers
- Tomahawk 6
- Broadcom Ethernet switch chip, 102.4 Tb/s with 512 lanes of 200G SerDes. Helios uses 2 per switch tray, 12 per rack. Broadcom
- TTFT
- Time to first token: mostly prefill time. Its partner metric is ITL, inter-token latency (decode).
- UALink
- Ultra Accelerator Link: open accelerator-to-accelerator interconnect standard (current spec: UALink 200G 1.0). Helios carries it over Ethernet (UALoE) through Tomahawk 6 switch trays. UALink Consortium
- UALoE
- UALink over Ethernet: AMD Helios' scale-up fabric, single switch hop through 6 in-rack Tomahawk 6 switch trays. AMD Helios brief
- UMC
- Unified Memory Controller: the AMD GPU block that drives the HBM stacks and owns HBM ECC. AMD RAS and ECC
- VCC
- Vector condition code: a 64-bit AMD scalar register pair that a vector compare (v_cmp_*) writes, one bit per lane; bits of lanes that are off are 0. Combined with EXEC to branch. How a GPU runs code
- VGPR
- Vector general-purpose register on AMD GPUs: one 32-bit value per lane of a wavefront. On MI355X a wave can use up to 512 (256 architectural + 256 accumulation); fewer per wave means more waves fit (occupancy). ROCm specs · Page
- VIP
- Virtual IP: an address that is not tied to one machine and can move between, or be shared by, several of them.
- vLLM
- An open-source LLM serving engine. Its engine runs as VLLM::EngineCore processes. vLLM docs
- Wedge
- A GPU state after a failed reset: the devices are still on PCIe but the GPU's boot processor (PSP) never came back, so software cannot use them. On MI35x only removing power (a cold power cycle) recovers it. GPU failures
- wgmma
- Hopper (sm_90a) PTX instruction: an asynchronous matrix multiply issued by a warpgroup of 128 threads, with operands in shared memory. PTX ISA · Page
- whereabouts
- Cluster-wide IPAM CNI plugin that stores IP allocations in Kubernetes CRDs (IPPool). A reconciler frees leaked addresses. GPU Pod networking · whereabouts
- wiping delay
- The 30 s the kubelet waits after a DRA driver disconnects before deleting that node's ResourceSlices for the driver (Kubernetes 1.35 defaultWipingDelay). kubelet DRA manager
- XCD
- Accelerator complex die: one of the 8 compute chiplets in an AMD MI300/MI350-series GPU. On MI355X each has 36 CUs (32 enabled) and a 4 MB L2. Compute partitions follow XCD lines. AMD CDNA 4 white paper · Learn: What a GPU is
- XCP
- Accelerated compute processor: the logical GPU produced by an AMD compute partition. SPX gives one XCP per card, CPX gives eight. Each XCP is enumerated as its own GPU by amd-smi and HIP. AMD SMI partitioning · Learn: Partitioning
- XGMI · also Infinity Fabric
- AMD’s GPU-to-GPU link (Infinity Fabric). The 8 GPUs of an MI355X server form one XGMI hive.
- XGMI hive
- The set of AMD GPUs joined by one coherent XGMI (Infinity Fabric) network; all 8 GPUs of an MI355X node. A fatal reset covers the whole hive. AMD tools · GPU failures
- Xid
- An NVIDIA driver error code printed to the kernel log, e.g. Xid 79 (GPU fell off the bus) or 48 (double-bit ECC). Xid catalog