Learn · GPUs in Kubernetes

DRA objects in detail: classes, slices, claims and the scheduler

Updated 10 Nov 2026 · 12 min read

GPU allocation in k8s gave the big picture. This page opens each DRA object, field by field, using illustrative example objects of a sample cluster (Kubernetes 1.35, AMD DRA driver v1.0.1). YAML is trimmed: managedFields, UIDs and resource versions are removed.

The idea

DRA is a small ordering system with four forms and one clerk. Each object answers one question:

Object Everyday picture Question it answers Who writes it
ResourceSlice a store’s shelf inventory What GPUs are on this node, and what is each like? the DRA driver
DeviceClass a product category What counts as “an AMD GPU”? the operator chart (admin)
ResourceClaimTemplate a blank order form What will each Pod of this workload need? the workload owner
ResourceClaim the filled-in order, with the item number stamped on it What does this Pod need, and which GPU did it get? controller (spec), scheduler (status)

The scheduler is the clerk: it walks the shelves in a fixed order and takes the first item that fits the category and the order (first fit). Click the i on a box for more.

Four objects, one clerk
The shelf inventory of one store: every GPU on one node, with its attributes. Written by the driver.ResourceSliceWhat is on this node?A category: a CEL filter over devices. Cluster-scoped, named, referred to by claims.DeviceClassWhat counts as an AMDGPU?A blank order form. Each Pod that names it gets its own filled-in copy.ResourceClaimTemplateWhat will each Pod need?The filled-in order. spec = what is needed; status = which device was picked and who uses it.ResourceClaimWhich GPU did this Podget?SchedulerWhich free GPU fits?one copy per Podinventoryfilterwrites the device
  1. The driver stocks the shelf: one ResourceSlice per node.
  2. The category says which shelf items count.
  3. Each Pod gets its own copy of the order form.
  4. The clerk finds a free match and stamps its ID on the order.

Where the picture breaks: nothing is shipped. The GPU stays in its slot; the Pod is sent to the GPU’s node. The stamp is a reservation that lasts as long as the claim. And the shelf list is rewritten by the driver whenever it likes, so editing it by hand does nothing.

Worked example: one order against one shelf

Take the ResourceSlice of gpu-node-1 and run a new one-GPU claim through the allocator by hand. Everything below is computed from the sample slice and claims, in the slice’s own device order, which is the order first fit walks.

A new claim meets gpu-node-1

1 · The shelf: 8 devices, in slice order

#deviceNUMAPCIpcieRootmemorynow
1gpu-3-13000000:4d:00.0pci0000:40294912Mifree
2gpu-4-13100000:6b:00.0pci0000:60294912Mifree
3gpu-5-13210000:8b:00.0pci0000:80294912Miheld
4gpu-6-13310000:a9:00.0pci0000:a0294912Miheld
5gpu-7-13410000:c7:00.0pci0000:c0294912Mifree
6gpu-8-13510000:e5:00.0pci0000:e0294912Mifree
7gpu-1-12800000:11:00.0pci0000:00294912Mifree
8gpu-2-12900000:2f:00.0pci0000:20294912Mifree

The driver discovered the GPUs in this order and published them as one slice (resourceSliceCount: 1). Two are already in claims: gpu-6-133 and gpu-5-132. Every GPU has its own PCIe root, while four GPUs share each NUMA node: 8 roots and 2 NUMA values for 8 devices.

2 · The class filter: device.driver == 'gpu.amd.com'

# DeviceClass gpu.amd.com
spec:
  selectors:
  - cel:
      expression: device.driver == 'gpu.amd.com'

The claim names deviceClassName: gpu.amd.com, so every device must pass the class’s selectors first. All 8 of 8 do: they were all published by that driver. A cluster with an NVIDIA driver too would see its devices fail here.

3 · The claim's own selector: ≥ 256 GiB

requests:
- name: gpu
  exactly:
    deviceClassName: gpu.amd.com
    count: 1
    selectors:
    - cel:
        expression: device.capacity["gpu.amd.com"].memory.compareTo(quantity("256Gi")) >= 0

Each device publishes memory: 294912Mi, that is 294912 ÷ 1024 = 288.0 GiB, so compareTo returns 1 and all 8 pass. Selectors are ANDed: class selectors and claim selectors must all be true.

4 · Skip what is taken

Match
8
class + selector
Held
2
gpu-6-133, gpu-5-132
Candidates
6

The allocator skips devices named in any other claim’s status.allocation, and devices held in memory for Pods in flight (the Reserve step), so two Pods scheduled at once cannot get the same GPU.

5 · First fit: gpu-3-130

status:
  allocation:
    devices:
      results:
      - {request: gpu, driver: gpu.amd.com, pool: gpu-node-1, device: gpu-3-130}
    nodeSelector:
      nodeSelectorTerms:
      - matchFields:
        - {key: metadata.name, operator: In, values: [gpu-node-1]}

The first free match in slice order is gpu-3-130 (row 1). There is no scoring: not “closest to the NIC”, not “least used”. The scheduler writes this in PreBind, then binds the Pod. Earlier Pods got other GPUs because other devices were free or held when they were placed.

6 · Ask for 2 GPUs on one PCIe root: Pending

requests:
- name: gpus
  exactly: {deviceClassName: gpu.amd.com, allocationMode: ExactCount, count: 2}
constraints:
- requests: [gpus]
  matchAttribute: resource.kubernetes.io/pcieRoot
pcieRootfree devices under it
pci0000:40gpu-3-130
pci0000:60gpu-4-131
pci0000:c0gpu-7-134
pci0000:e0gpu-8-135
pci0000:00gpu-1-128
pci0000:20gpu-2-129

A constraint says all devices of the listed requests must share one attribute value. Here the largest group has 1 device, so no pair exists on this node. The same holds on every node of this design: each slice has 8 GPUs under 8 different roots. The Pod stays Pending with a “cannot allocate all claims” event. A selector or constraint that looks reasonable can be impossible on the real hardware: read the slice first.

Device order, attributes and held devices are example output in the shape of kubectl get resourceslice / resourceclaims on the sample cluster. The selector results are computed on this page with the same logic as the CEL shown.

The identity of a device

DRA names a device with three strings: (driver, pool, device).

  • driver: gpu.amd.com. One DRA driver, one name, in DNS-subdomain form (max 63 characters).
  • pool: a group of devices the driver manages together. The AMD driver uses one pool per node, named after the node: gpu-node-1.
  • device: unique inside the pool. The AMD driver uses gpu-<card>-<render>, from the GPU’s DRM device numbers: gpu-5-132 is /dev/dri/card5 plus /dev/dri/renderD132.

The triple is what a claim records. It is stable only as long as the driver names devices the same way. The PCI address in the resource.kubernetes.io/pciBusID attribute ties a device to a physical slot, which is what you want for hardware tickets.

DeviceClass

A DeviceClass is a cluster-scoped, named filter over devices. Admins or drivers create it; workloads refer to it by name. The AMD operator chart creates this one:

apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
  name: gpu.amd.com
  labels:
    app.kubernetes.io/managed-by: Helm
    helm.sh/chart: gpu-operator-charts-v1.5.1
spec:
  selectors:
  - cel:
      expression: device.driver == 'gpu.amd.com'
  • spec.selectors: CEL expressions. A device belongs to the class only if all of them are true.
  • spec.config (not used here): opaque, driver-specific parameters passed to the driver for every device allocated through this class.
  • Being cluster-scoped, a class is also a policy point. An admin can create a narrower class, for example mi355x-whole with an extra selector on productName and type, and allow only that class in some namespaces through admission policy.

ResourceSlice

A ResourceSlice is the driver’s inventory. The driver owns it; any manual edit is overwritten. The AMD driver writes one slice per node, owned by the Node object, so deleting the Node deletes the slice.

apiVersion: resource.k8s.io/v1
kind: ResourceSlice
metadata:
  name: 00000-gpu.amd.com-gpu-node-1-x7q2m
  generateName: 00000-gpu.amd.com-gpu-node-1-
  ownerReferences:
  - {apiVersion: v1, kind: Node, name: gpu-node-1, controller: true}
spec:
  driver: gpu.amd.com
  nodeName: gpu-node-1          # only this node can use these devices
  pool:
    name: gpu-node-1
    generation: 1                    # bumped when the pool changes
    resourceSliceCount: 1            # slices that make up this pool
  devices:
  - name: gpu-5-132
    attributes:
      type:              {string: amdgpu}               # full GPU, not a partition
      productName:       {string: Instinct_MI355X_OAM}
      deviceID:          {string: "0x75a3"}             # PCI device ID
      driverVersion:     {version: 6.14.0}              # semver type
      driverVersionFull: {string: 6.14.0.70000}
      partitionProfile:  {string: spx_nps1}
      numaNode:          {int: 1}
      resource.kubernetes.io/pciBusID: {string: "0000:8b:00.0"}
      resource.kubernetes.io/pcieRoot: {string: pci0000:80}
    capacity:
      memory:       {value: 294912Mi}    # 288 GiB of HBM
      computeUnits: {value: "256"}
      simdUnits:    {value: 1Ki}
  # … 7 more devices

Field notes:

  • Node access. Exactly one of nodeName, nodeSelector, allNodes or perDeviceNodeSelection says which nodes can reach the devices. GPUs are local, so nodeName. A network-attached device could use allNodes: true.
  • Pools and generations. A large pool may span several slices. The scheduler uses a pool only when it sees all resourceSliceCount slices with the newest generation. A half-published pool is ignored, which prevents allocation from a stale inventory.
  • Typed attributes. Each attribute is one of string, int, bool or version (semver). driverVersion is a version, so CEL can compare it with semver(). Unqualified names (numaNode) belong to the driver’s domain. Qualified names (resource.kubernetes.io/pciBusID) are shared, standard attributes so different drivers can be aligned.
  • Capacities are Kubernetes quantities. 294912Mi is 294912 ÷ 1024 = 288 GiB; the node label says 288G.
  • Limits. A slice holds at most 128 devices (64 if it uses taints or shared counters). Each device has at most 32 attributes plus capacities.
  • The 00000- prefix. The scheduler is first fit and walks slices and pools in name order. The kubeletplugin helper library prefixes names so that the order matches the driver’s wish.

The driver logs the same content when it publishes. In example output, a fresh driver Pod on gpu-node-3 prints Discovered 8 AMD GPU devices and then Publishing ResourceSlice: {"Pools": {"gpu-node-3": …}}.

ResourceClaimTemplate

A ResourceClaimTemplate is a namespaced blueprint. Each Pod that references it gets its own claim. The template for the prefill model:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: llm-prefill-gpu
  namespace: gpu-demo
  labels:
    example.com/model-id: gpt-oss-120b
spec:
  spec:                         # the ResourceClaim spec to stamp out
    devices:
      requests:
      - name: gpu               # request name; referenced in results
        exactly:
          deviceClassName: gpu.amd.com
          allocationMode: ExactCount
          count: 1

And the Pod side, from the Deployment’s Pod template:

spec:
  resourceClaims:
  - name: gpu                                           # name inside the Pod
    resourceClaimTemplateName: llm-prefill-gpu
  containers:
  - name: prefill
    resources:
      claims: [{name: gpu}]                             # this container uses it
      limits: {cpu: "8", memory: 256Gi, rdma/hca_devices: "1"}

spec.resourceClaims declares the claim for the Pod; resources.claims says which containers see the devices. A container that does not list the claim gets no device files.

Request forms

Field Meaning Status in 1.35
exactly.allocationMode: ExactCount, count: N N devices that match GA
exactly.allocationMode: All every matching device on one node; fails if any is in use GA
exactly.selectors[].cel extra filter on top of the class GA
exactly.adminAccess: true access devices even if in use (see below) beta, on
firstAvailable: [...] prioritized list: try sub-request 1, else 2, … (max 8) beta, on
constraints[].matchAttribute all devices in the listed requests share an attribute value GA
config[].opaque driver-specific parameters for some requests GA

A claim can hold at most 32 requests, 32 constraints and 32 allocated devices.

CEL selectors

CEL (Common Expression Language) is the same language Kubernetes uses in ValidatingAdmissionPolicy. In DRA, each expression sees one object, device, with driver, attributes and capacity. Attributes and capacities are grouped by domain.

selectors:
# only whole GPUs, not partitions
- cel: {expression: 'device.attributes["gpu.amd.com"].type == "amdgpu"'}
# at least 256 GiB of memory
- cel: {expression: 'device.capacity["gpu.amd.com"].memory.compareTo(quantity("256Gi")) >= 0'}
# driver new enough
- cel: {expression: 'device.attributes["gpu.amd.com"].driverVersion.isGreaterThan(semver("6.12.0"))'}
# a fixed NUMA node
- cel: {expression: 'device.attributes["gpu.amd.com"].numaNode == 1'}

Two GPUs on the same PCIe root, written as a constraint:

requests:
- name: gpus
  exactly: {deviceClassName: gpu.amd.com, allocationMode: ExactCount, count: 2}
constraints:
- requests: [gpus]
  matchAttribute: resource.kubernetes.io/pcieRoot

Try a selector on the sample slice

Pick a selector to see which of the 8 devices on gpu-node-1 it matches, and which one first fit would allocate right now.

Selector
- cel:
    expression: device.capacity["gpu.amd.com"].memory.compareTo(quantity("256Gi")) >= 0
deviceNUMAPCImatchnow
gpu-3-13000000:4d:00.0✓first fit: allocated
gpu-4-13100000:6b:00.0✓free
gpu-5-13210000:8b:00.0✓held
gpu-6-13310000:a9:00.0✓held
gpu-7-13410000:c7:00.0✓free
gpu-8-13510000:e5:00.0✓free
gpu-1-12800000:11:00.0✓free
gpu-2-12900000:2f:00.0✓free

8 of 8 match; 6 of those are free. First fit picks gpu-3-130.

Each expression is capped at 10 KiB and a runtime cost of 1,000,000 units, roughly 0.1 s per evaluation on the reference machine.

ResourceClaim: an allocated one

The resourceclaim controller stamped this claim out of the template for Pod …-p4x7k. The scheduler filled in status.

apiVersion: resource.k8s.io/v1
kind: ResourceClaim
metadata:
  name: llm-prefill-5f7d8c9b6-p4x7k-gpu-z8k3q
  namespace: gpu-demo
  annotations:
    resource.kubernetes.io/pod-claim-name: gpu         # name inside the Pod
  finalizers:
  - resource.kubernetes.io/delete-protection           # no delete while in use
  ownerReferences:
  - {apiVersion: v1, kind: Pod, name: llm-prefill-5f7d8c9b6-p4x7k, controller: true}
spec:                                                  # copied from the template
  devices:
    requests:
    - name: gpu
      exactly: {deviceClassName: gpu.amd.com, allocationMode: ExactCount, count: 1}
status:
  allocation:
    devices:
      results:
      - request: gpu
        driver: gpu.amd.com
        pool: gpu-node-1
        device: gpu-5-132
    nodeSelector:                                      # where the Pod may run
      nodeSelectorTerms:
      - matchFields:
        - {key: metadata.name, operator: In, values: [gpu-node-1]}
  reservedFor:                                         # who uses it now
  - {resource: pods, name: llm-prefill-5f7d8c9b6-p4x7k, uid: b0d10001-…}
  • status.allocation.devices.results: the source of truth for “who has which GPU”.
  • status.allocation.nodeSelector: copied from the slice’s node access. Any Pod that uses this claim must run on gpu-node-1.
  • status.reservedFor: consumers, max 256. A claim can be shared by several Pods if they all reference it by name (resourceClaimName instead of a template). Template claims are private to one Pod.
  • The Pod links back in status.resourceClaimStatuses: [{name: gpu, resourceClaimName: …-gpu-z8k3q}].
  • kubectl get resourceclaims prints STATE allocated,reserved: allocated means status.allocation is set, reserved means reservedFor is not empty.
  • status.devices (not used by the AMD driver) lets a driver report per-device conditions or network data. The feature gate DRAResourceClaimDeviceStatus is beta and on in 1.35.

How the scheduler allocates

From Pod creation to an allocated claim
SCHEDULING CYCLECLAIM CREATEDBINDING CYCLEReplicaSet controllerAPI serverresourceclaim controllerScheduler(DynamicResources)kubeletcreate Pod(resourceClaimTemplateName)watch: Pod needs a claimcreate ResourceClaim, owner = Pod; setPod resourceClaimStatusesPod schedulable (PreEnqueue passes)PreFilter + Filter: which nodes havematching free devicesReserve: pick gpu-5-132, hold inmemoryPreBind: write status.allocation +reservedForBind Pod to gpu-node-1Pod assigned: NodePrepareResourcesnext
Names are from the sample prefill Pod. The scheduler phases are the DynamicResources plugin's extension points. Press play to step through.
  1. PreEnqueue. The Pod is not even tried until every claim it references exists. A Pod whose template is missing waits here.
  2. PreFilter. The plugin gathers the Pod’s claims. Claims already allocated (shared claims) fix the node. Unallocated claims are collected for allocation.
  3. Filter (per node). For each candidate node, the plugin runs the allocator over the slices that node can reach. It tries devices in slice and pool name order, checks class selectors, claim selectors and constraints, and skips devices already allocated to other claims or held by Pods in flight. First fit wins; there is no “best GPU” scoring in the base algorithm.
  4. Reserve. The chosen devices go into an in-memory “in flight” set, so a second Pod scheduled in parallel cannot pick them.
  5. PreBind. The plugin writes status.allocation and adds the Pod to reservedFor in the API. If that write fails, Unreserve drops the in-memory hold and the Pod is retried.
  6. Bind. Only now does the Pod get spec.nodeName.

Things this implies:

  • Normal scheduling rules still apply. CPU, memory, node affinity, taints and rdma/hca_devices filter nodes before the device check matters. A Pod with a perfect GPU match still fails on a node without 256 GiB of free memory.
  • No preemption for devices. The scheduler does not evict lower-priority Pods to free a GPU. A full cluster leaves the Pod Pending.
  • Bypassing the scheduler breaks DRA. A Pod created with spec.nodeName already set skips allocation. The kubelet refuses to start it until its claims are allocated and reserved for it.
  • Load. Every candidate node runs CEL for every device. In a cluster of a few hundred GPUs that is small. The alpha gate DRASchedulerFilterTimeout (off in 1.35) can cap a slow filter.

Admin access

A request with adminAccess: true can be allocated a device that is already in use by another claim. It exists for monitoring and debugging agents that must see every GPU. The canonical form uses allocationMode: All to get every matching device on a node.

  • Only allowed in namespaces labelled resource.kubernetes.io/admin-access: "true". The API server rejects it elsewhere.
  • Feature gate DRAAdminAccess: beta, on by default in 1.34 and 1.35, GA in 1.36.
  • The AMD metrics exporter does not need it. It reads GPUs through the host and maps them to Pods through the kubelet’s PodResources API.

Newer features and their status

Prioritized list (firstAvailable). “Give me one MI355X, else two smaller GPUs.” The scheduler tries sub-requests in order. Beta, on in 1.35. Useful for mixed fleets; a single-model fleet does not need it.

Partitionable devices. Lets a driver publish overlapping devices (a whole GPU and its partitions) that consume from shared counters, so allocating a partition makes the whole GPU unavailable and the other way round. Alpha, off in 1.35 (DRAPartitionableDevices), beta from 1.36. The AMD driver v1.0.1 does not create partitions; it publishes whatever mode the GPU is already in (type: amdgpu or amdgpu-partition). See GPU partitioning.

Device taints and DeviceTaintRule. Like node taints, but on one device. Effects: NoSchedule (do not allocate), NoExecute (also evict Pods using it, done by a controller in kube-controller-manager) and None (informational). Drivers can put taints in slices (DRADeviceTaints); admins can taint devices without the driver through a cluster-scoped DeviceTaintRule (DRADeviceTaintRules, served as resource.k8s.io/v1alpha3 in 1.35). Both are alpha and off in 1.35. A cluster that has not enabled the gate does not serve the API (kubectl get devicetaintrules fails). An alpha API carries no upgrade guarantee, so anything built on it must expect changes at 1.36.

Consumable capacity. Several claims share one device and each consumes part of a capacity (for example memory). Alpha in 1.35.

Extended resource mapping. A DeviceClass can declare extendedResourceName, so legacy Pods asking for amd.com/gpu: 1 are served by DRA. Alpha in 1.35, beta in 1.36.

Binding conditions. Delay binding until a device reports ready (for attachable fabric devices). Alpha in 1.35.

Feature gates in Kubernetes 1.35

From the Kubernetes feature-gate reference. “On” means on by default; nothing on the sample cluster overrides the defaults (no --feature-gates on the API server, scheduler or controller-manager, none in the kubelet config).

Gate Stage in 1.35 Default What it unlocks
DynamicResourceAllocation GA (since 1.34) on the core API and scheduling
DRAAdminAccess beta on adminAccess requests
DRAPrioritizedList beta on firstAvailable
DRAResourceClaimDeviceStatus beta on status.devices
KubeletPodResourcesDynamicResources beta on DRA devices in the PodResources API
DRAPartitionableDevices alpha off shared counters
DRADeviceTaints alpha off taints in slices, tolerations in claims
DRADeviceTaintRules alpha off DeviceTaintRule objects
DRAConsumableCapacity alpha off shared consumable capacity
DRAExtendedResource alpha off extended-resource mapping
DRADeviceBindingConditions alpha off binding conditions
DRASchedulerFilterTimeout alpha off filter time limit
ResourceHealthStatus alpha off device health in Pod status

Alpha gates must be turned on in each component that uses them (API server, scheduler, controller-manager, kubelet), and alpha API versions need --runtime-config on the API server.

Read it yourself

kubectl get deviceclass gpu.amd.com -o yaml
kubectl get resourceslices -o wide
kubectl get resourceslices --field-selector spec.nodeName=gpu-node-1 -o yaml
kubectl get resourceclaimtemplates -A
kubectl get resourceclaims -A -o wide
kubectl get resourceclaim <claim> -n <ns> -o jsonpath='{.status.allocation.devices.results}'
# which Pod holds which GPU, cluster-wide
kubectl get resourceclaims -A -o json | jq -r '.items[] | [.metadata.namespace, (.status.reservedFor[0].name // "-"), (.status.allocation.devices.results[0] | "\(.pool)/\(.device)")] | @tsv'

Misconceptions

  • “The DeviceClass decides which GPUs a namespace may use.” Only if you build that. By default any namespace may create claims against any class. Access control is RBAC on resourceclaims plus admission policy.
  • “Deleting a claim frees the GPU at once.” The delete-protection finalizer keeps the claim while it is reserved. The GPU is free after the Pod stops and the kubelet has unprepared it.
  • “Editing a ResourceSlice fixes bad data.” The driver overwrites it. Fix the driver or the node.
  • “The pool name is the node name.” For the AMD driver, yes. It is the driver’s choice, not a rule.

Further reading