This canvas covers the shared GPU platform that physical-AI work for heavy industry (construction, mining and earth-moving autonomy) runs on: GPU Kubernetes, Ray, queues, quotas, scheduling, networking and storage for training, and the infrastructure-as-code that keeps it reproducible. It is canvas 6 of 9 and follows a research-area exploration framework, with the Emerging (11–12) and Applied (13) modules added.
| Boundary | Owned here | Owned elsewhere |
|---|---|---|
| Training jobs | Where and when jobs run; quotas; preemption; multi-node placement | Parallelism strategy and model code (canvas 3) |
| Simulation | Scheduling render, generation and RL rollout jobs | Simulator content and physics (canvas 2) |
| Evaluation | Reserved capacity for release gates | Gate design (canvas 4) |
| Inference | Cluster capacity for cloud-side serving | Engine optimization and serving stack (canvas 7); on-machine compute (edge) |
| Cost and SLOs | Exposing utilization and allocation data | Standards, chargeback, SLO targets (canvas 8) |
Scope note: physical cluster and network wiring design (racks, power, cooling, NVLink or InfiniBand fabric layout) is discussed here only at the level of workload requirements and conceptual constraints, not as a design.
Definition. GPU compute orchestration is the layer that decides which AI workload runs on which accelerators, when, and with what share, across teams and job types. In 2026 the consensus substrate is Kubernetes: GPU scheduling, KubeRay for distributed training, multi-tenancy controls and DRA-based allocation are supported by a mature ecosystem (CloudOptimo).
Why it matters for heavy-industry physical AI. GPUs are the most expensive and contested resource in a physical-AI platform. Autonomy for earth-moving, haulage and site work needs several workload types at once (fleet-data processing, simulation and rendering, reinforcement learning, large model training, release-gate evaluation, cloud inference), and they have different scheduling needs. Without queues, quotas and gang scheduling, teams hoard capacity, large jobs starve and utilization collapses.
Platform layers (bottom to top): GPU nodes and networking → Kubernetes with the GPU Operator and Dynamic Resource Allocation (DRA) → admission and scheduling (Kueue, KAI Scheduler) → compute frameworks (Ray via KubeRay, PyTorch jobs, Isaac Lab) → workloads from canvases 1–4 and 7.
The queue layer is highlighted because it is where fairness, priority and utilization are decided; everything below it can be swapped without changing how teams submit work.
Key sub-domains
| Sub-domain | Core question | Typical artifacts |
|---|---|---|
| Hardware and networking | Which GPUs, interconnects and storage for which jobs? | Node pools; InfiniBand or RoCE fabric; parallel file systems (workload requirements only here) |
| Device management | How are GPUs exposed and shared? | GPU Operator, DRA drivers, MIG and time-slicing configs |
| Admission and queues | Which job may start now, under whose quota? | Kueue ClusterQueues and LocalQueues; priority classes |
| Scheduling and placement | Where should pods land for performance? | Gang scheduling; topology-aware placement; bin packing |
| Compute frameworks | How do distributed jobs run? | KubeRay (RayJob, RayCluster, RayService), Kubeflow Trainer, JobSet |
| Hybrid capacity | When to burst to cloud GPUs? | Multi-cluster queues; cloud node pools from GPU clouds and hyperscalers |
| Infrastructure as code | Can the platform be rebuilt from code? | Terraform, Helm, GitOps (Argo CD), policy-as-code |
| Reliability | How do long jobs survive failures? | Checkpointing, node health checks, automatic restart |
Debate. Kubernetes versus Slurm. HPC teams prefer Slurm for large training; platform teams prefer Kubernetes for everything else. Kubernetes is closing the gap (gang scheduling is now beta in core Kubernetes 1.37, per the cloudmagazin report cited in section 2), and hybrids such as Slurm-on-Kubernetes exist.
GPU orchestration moved from HPC batch schedulers, to Kubernetes treating GPUs as opaque counts, to a scheduler that understands GPU topology, slices and gangs.
| Year | Milestone | Why it matters here |
|---|---|---|
| 2026 | Kubernetes 1.37: Workload and PodGroup APIs with all-or-nothing gang scheduling in beta (cloudmagazin) | Gang scheduling enters core Kubernetes; Kueue still handles quotas |
| 2026 | KubeCon Europe: NVIDIA donates DRA driver to CNCF; KAI Scheduler accepted as CNCF Sandbox project (Rafay) | GPU scheduling becomes community-owned |
| 2025 | DRA GA in Kubernetes 1.34 (Techplained); KubeRay 1.5 with Kueue autoscaler and KAI integration (Anyscale) | Declarative GPU claims; Ray integrated with Kubernetes queues |
| 2024 | NVIDIA acquires Run:ai; later open-sources the KAI Scheduler | Commercial GPU scheduling ideas move upstream |
| 2022–2023 | Kueue started (Kubernetes SIG); GPU clouds scale | Kubernetes-native queues; elastic GPU capacity |
| 2020–2021 | NVIDIA A100 with MIG; GPU Operator; KubeRay started | Hardware partitioning; automated GPU node setup |
| 2017–2019 | Kubernetes device plugins (v1.8); Kubeflow; Ray (2017); Volcano | GPUs schedulable in Kubernetes; ML on Kubernetes |
| 2000s–2010s | Slurm, PBS on HPC clusters; YARN | Batch queues, fair share, gang allocation for MPI jobs |
Industry context (a different clock). Industrial autonomy itself predates this scheduling stack: Komatsu describes its FrontRunner Autonomous Haulage System as commercially deployed since 2008 (Komatsu), and Rio Tinto has taken delivery of its 300th Komatsu autonomous haul truck (alcircle; company-reported, date not confirmed). Those systems are rule-and-sensor based; the newer learned-model approaches (canvas 3) are what create demand for the GPU platform described here. That link is an interpretation, not a sourced finding.
Paradigm shifts
From counts to devices. The device plugin exposed GPUs as integers; DRA lets jobs claim devices with attributes and topology (Spheron).
From pods to gangs. Distributed jobs need all workers at once or none.
From static clusters to queues with borrowing. Teams get guaranteed quotas and borrow idle capacity, with preemption to give it back.
From one cluster to fleets of clusters. On-premises plus GPU clouds, with work dispatched across them.
The tools are mature; the problem is utilization. Enterprise GPU clusters often run far below capacity because teams over-reserve and jobs fit badly on the hardware.
Dominant frameworks
Kueue for admission and quotas. Kueue sits in front of the scheduler, deciding when jobs start under multi-tenant quotas, and natively handles Job, JobSet, KubeRay, Kubeflow Trainer and LeaderWorkerSet (Kueue). Its objects are ClusterQueues (quotas and fair sharing), LocalQueues (per-team) and ResourceFlavors (hardware types) (Ray docs).
KAI Scheduler for AI-aware placement. Gang scheduling, hierarchical queues with Dominant Resource Fairness, and pre-scheduling simulation to cut preemption costs (Pulse / Rafay).
DRA for device allocation. Pods claim devices through ResourceClaims; topology (NUMA, NVLink) and MIG integrate with scheduling (Rafay).
Ray on Kubernetes. KubeRay with Kueue gives gang and priority scheduling for RayJob, RayCluster and RayService (Ray docs). Apple runs a similar multi-tenant Ray platform on YuniKorn (ZenML).
GPU sharing. MIG gives isolated slices; time-slicing shares a whole GPU when isolation isn't needed (arXiv 2602.04900).
Key research questions
| Question | State of evidence |
|---|---|
| Why is utilization so low? | A 2026 report on about 23,000 clusters found 5% average GPU utilization (Cast AI; vendor report); some well-run clusters reach 30–49% (TechRadar) |
| Does GPU sharing pay off? | MIG with DRA on H100 cut required GPUs from 100 to 25–75 for the same workloads in Cast AI's data (Cast AI blog; vendor data); at Alibaba, sharing is rarely used and fragmentation is now negligible (OSDI 2026) |
| How to schedule heterogeneous fleets? | Alibaba's 2026 trace shows mixed GPU generations and workloads as the norm (OSDI 2026 paper) |
| How to share between training, simulation, RL and inference? | Priority classes plus preemption; RL needs co-scheduled rollout and learner pools (practitioner pattern, little published evidence for physical AI) |
| When does the network, not the GPU, limit throughput? | As clusters grow, interconnect becomes the binding constraint (Chip on my Shoulder; commentary) |
Challenges specific to heavy-industry physical AI
Mixed workloads. Simulation needs RTX-class GPUs for rendering; training needs datacenter GPUs; inference needs low-latency nodes.
Hybrid capacity. Organizations combine on-premises clusters with GPU clouds; data gravity and egress costs shape placement. Fleet video and sensor data from jobsites and mines is large, which makes data location a first-order constraint (canvas 1).
Many small teams. Autonomy, perception, operator-assistance and digital-twin efforts all compete for the same GPUs.
Debates
Kueue vs KAI Scheduler vs Volcano/YuniKorn: overlapping features; many platforms combine Kueue for quotas with KAI or the core scheduler for placement.
Own the GPUs vs rent from GPU clouds: utilization decides the economics.
Fractional GPUs vs whole-GPU allocation: sharing helps inference and notebooks, rarely large training.
NVIDIA and the CNCF community shape the software layer; GPU clouds and hyperscalers shape capacity; systems researchers publish what production clusters really look like.
| Cluster | Who | Contribution to this canvas |
|---|---|---|
| NVIDIA | GPU Operator, DRA driver, KAI Scheduler (from Run:ai), Mission Control, DGX SuperPOD reference architectures (NVIDIA) | Reference designs for AI factories; open-sourced scheduling components |
| Kubernetes community | SIG Scheduling, WG Batch, WG Device Management (Kueue, JobSet, DRA, gang scheduling) | Core APIs for AI workloads |
| Other schedulers | Volcano (CNCF), Apache YuniKorn, Slurm (SchedMD, now NVIDIA), Slinky (Slurm on Kubernetes) | Batch and HPC scheduling |
| Ray ecosystem | Anyscale (Ray, KubeRay) | Distributed Python and RL frameworks |
| GPU clouds and hyperscalers | Azure, AWS, GCP, Nebius, Lambda; CoreWeave as one example of a GPU cloud with Kueue support (docs) | Burst and dedicated capacity; managed Kubernetes |
| Platform vendors | Rafay, Run:ai (NVIDIA), Red Hat OpenShift AI, Cast AI | Packaged multi-tenant GPU platforms and optimization |
| Systems research | Alibaba (OSDI 2026 cluster trace), Microsoft (Philly, Singularity), Google (Borg), UC Berkeley Sky Computing, CMU | Cluster characterization and scheduling research |
Inside heavy-equipment OEMs and autonomy developers (public evidence). Public evidence on how industrial autonomy developers run their GPU infrastructure is thin; most disclose products and deployments, not cluster designs. What is documented:
Mining autonomy operators and suppliers. Komatsu describes FrontRunner autonomous haulage as commercially deployed since 2008 (Komatsu); Rio Tinto reports Komatsu autonomous trucks across its Australian operations, with its 300th delivered (alcircle). Neither source says anything about training-compute infrastructure.
Construction autonomy startups. Bedrock Robotics reports autonomous excavator deployments on commercial sites, with software running on an appliance placed in the excavator cab; NVIDIA's venture arm is a reported investor (Electrek; BD+C). No source found names its training infrastructure.
Frameworks and clouds. The shared stack for this industry is mostly the general AI stack: NVIDIA reference architectures, Kubernetes, Ray and GPU clouds, as documented in the tables above. The industry-specific unknown is how these actors split workloads between owned and rented capacity.
Debate. Buy a packaged platform (Run:ai, Rafay, OpenShift AI) or assemble upstream components (Kueue, KAI, KubeRay)? Packaged is faster; upstream avoids licence cost and lock-in.
Methods group by concern. Each has a proven default and a frontier.
| Concern | Default method | Frontier |
|---|---|---|
| Quotas and fairness | Hierarchical queues per org and team; guaranteed quota plus borrowing; fair sharing | Dominant Resource Fairness across GPU, CPU, memory and network (KAI) |
| Priority and preemption | Priority classes (release gates > production retraining > research > notebooks); preempt borrowed capacity first | Simulation-based preemption that predicts cost before evicting |
| Gang scheduling | Kueue or KAI admits all pods of a job together; core Kubernetes 1.37 PodGroups in beta | Elastic gangs that grow and shrink with capacity |
| Topology-aware placement | Place multi-node jobs inside one NVLink domain or network block | DRA attributes for NVLink and NIC affinity (Spheron) |
| GPU sharing | MIG slices for inference and notebooks; time-slicing for development | Dynamic MIG reconfiguration by demand |
| Hybrid and multi-cluster | Separate clusters, manual placement | Multi-cluster queues (Kueue MultiKueue) dispatching to on-prem or cloud |
| Reliability | Checkpoint every N minutes; health checks; automatic restart | Fast in-memory checkpointing; predictive node-failure detection |
| Infrastructure as code | Terraform for cloud and network; Helm charts; Argo CD GitOps; policy-as-code (OPA, Kyverno) | Versioned platform blueprints rolled out across clusters (Pulse / Rafay) |
| Isolation and security | Namespaces, RBAC, network policies | Kata Containers for VM-level isolation of shared GPUs |
Workload classes to define
| Class | Example | Scheduling needs |
|---|---|---|
| Large training | VLA post-training, world-model fine-tuning | Gang, multi-node, topology-aware, checkpointing |
| RL | Isaac Lab RL, VLA RL | Co-scheduled rollout (RTX or datacenter) and learner pools |
| Simulation and rendering | Synthetic data generation, sim-to-real transfer, log replay | RTX GPUs; high-throughput batch |
| Data processing | Curation, auto-labeling, embeddings (canvas 1) | Batch, preemptible, often Ray |
| Evaluation gates | Release suites (canvas 4) | Reserved capacity before releases |
| Inference (cloud) | Operator assistants, evaluation-time teacher models | Low latency, autoscaling, MIG |
| Interactive | Notebooks, debugging | Small slices, idle timeouts |
Analysis techniques
Allocation versus actual utilization per queue (from DCGM metrics).
Queue wait time and job completion time per class.
Goodput: useful training progress per GPU-hour after failures and restarts.
Every other canvas consumes this platform. Each interface is a workload class with a contract.
| Connects to | What crosses the boundary | Contract to define |
|---|---|---|
| Canvas 1: data flywheel | Curation, labeling and embedding jobs | Preemptible batch class; storage locality rules |
| Canvas 2: simulation | Rendering, generation and RL rollout jobs | RTX node pools; batch priority |
| Canvas 3: training | Multi-node training and RL learner jobs | Gang scheduling, topology placement, checkpoint storage |
| Canvas 4: evaluation | Release-gate suites | Reserved capacity windows |
| Canvas 5: MLOps | Pipelines that trigger jobs | Service accounts, quotas for automated pipelines |
| Canvas 7: inference | Cloud-side serving | Inference node pools; autoscaling; MIG |
| Canvas 8: observability and cost | DCGM metrics, allocation records | Labels for team, project and workload class on every job |
| Canvas 9: reference architecture | Platform blueprint reused across sites and clouds | One versioned blueprint for on-prem and cloud clusters |
Outside fields
Data-center engineering: power, liquid cooling and floor space for dense GPU racks (conceptual constraint only here).
Networking: InfiniBand or Ethernet (Spectrum-X) fabrics; storage networks (conceptual constraint only here).
Finance and procurement: buy versus rent; reserved versus on-demand cloud capacity.
Security: multi-tenant isolation, supply-chain security for containers and drivers.
Operations research: scheduling theory and queueing models.
GPU scheduling is converging into core Kubernetes, while the hard problems shift to heterogeneity, networks and economics. The items below are trends reported in the cited sources, not forecasts of outcomes.
Emerging trends
Scheduling in the core. DRA is GA and gang scheduling is beta in Kubernetes; add-ons focus on quotas, fairness and multi-cluster dispatch.
Kubernetes and Slurm converge. NVIDIA acquired SchedMD in December 2025 and committed to keep Slurm open source and vendor-neutral (Converge Digest); Slurm-on-Kubernetes projects bridge the two.
Heterogeneous fleets. Multiple GPU generations and types in one cluster are now normal (OSDI 2026).
Network-aware scheduling. Placement by NVLink domain and network block matters as much as GPU count.
AI-factory blueprints. Reference designs cover power, cooling and digital twins of the data center itself (Data Center Frontier).
Utilization as a finance metric. With average GPU utilization reported at 5% in one vendor survey (Cast AI, above), finance teams are reported to be asking for it (TechRadar).
Research gaps in heavy-industry physical AI
| Gap | Why it is open | Opportunity |
|---|---|---|
| Co-scheduling simulation, rendering and RL | Most schedulers treat jobs independently | Workload class for RL that co-places rollout and learner pools |
| Mixed RTX and datacenter GPU fleets | Few published policies for graphics-plus-compute clusters | Flavor-based queues by GPU type with borrowing rules |
| Hybrid on-prem and GPU-cloud dispatch | Data gravity and egress costs complicate placement | Multi-cluster queues with data-location awareness |
| Fair share across many small teams | Static quotas waste capacity; free-for-all starves teams | Borrowing with preemption and clear priority classes |
| Published cluster traces for physical AI | Public traces (Alibaba, Philly, Borg) are not from robotics or autonomy workloads | Open traces mixing fleet-data processing, simulation, RL and evaluation jobs |
Potential breakthroughs
Sustained utilization well above reported averages through queues, borrowing and right-sized GPU slices (a hypothesis to test, not a demonstrated result for physical AI).
One platform blueprint that deploys the same stack to on-premises clusters and to cloud GPUs.
Tools and platforms, by layer
| Layer | Open source / standards | Commercial / vendor |
|---|---|---|
| Hardware reference | — | NVIDIA DGX SuperPOD and BasePOD reference architectures; certified storage (NVIDIA) |
| Device management | NVIDIA GPU Operator, DRA driver (CNCF), DCGM exporter | NVIDIA Mission Control |
| Admission and quotas | Kueue (incl. MultiKueue), Volcano, YuniKorn | Run:ai, Rafay |
| Scheduling | Kubernetes scheduler (gang beta in 1.37), KAI Scheduler (CNCF Sandbox), Slurm, Slinky | — |
| Frameworks | KubeRay, Kubeflow Trainer, JobSet, LeaderWorkerSet | Anyscale |
| Infrastructure as code | Terraform / OpenTofu, Helm, Argo CD, Flux, Crossplane, OPA / Kyverno | Rafay blueprints |
| Managed GPU Kubernetes | — | AKS, EKS, GKE, Nebius, GPU clouds such as CoreWeave |
| Cost and optimization | OpenCost | Cast AI, Kubecost |
Journals and conferences
Systems: OSDI, SOSP, NSDI, EuroSys, ATC, MLSys.
HPC: SC, ISC, IPDPS.
Cloud native: KubeCon + CloudNativeCon (AI and batch tracks); Ray Summit.
Industry: NVIDIA GTC.
Public datasets
Cluster traces: Alibaba (OSDI 2026 heterogeneous AI trace), Microsoft Philly, Google Borg traces.
Funding sources
Corporate infrastructure budgets and capital spend on GPU hardware; cloud commitments (reserved capacity discounts).
Vendor partner programs (NVIDIA, hyperscalers, GPU clouds).
Venture funding for autonomy developers; for example, Bedrock Robotics reported an $80M raise in 2025 (Electrek).
Read the Alibaba OSDI 2026 paper for how real clusters behave, then the Kueue and KAI documentation for practice.
Recent influential works (2024–2026)
| Year | Work | One-line takeaway |
|---|---|---|
| 2026 | Li et al., Heterogeneity at Hyperscale (OSDI, Alibaba) | Heterogeneous fleets are the norm; fractional-GPU fragmentation now negligible because sharing is rare |
| 2026 | Kueue, DAS and GAIE on Kubernetes (arXiv 2602.04900) | Kubernetes components together form a full AI training and serving platform |
| 2026 | Cast AI, State of Kubernetes Optimization | 5% average GPU utilization in non-optimized clusters (vendor report) |
| 2026 | NVIDIA KubeCon contributions: DRA driver, KAI, Kata (Rafay) | GPU scheduling moves to community ownership |
| 2025 | Apple, Elastic GPU management for Ray with YuniKorn | Production pattern for multi-tenant Ray with quotas and preemption |
| 2025 | KubeRay v1.5 | Kueue autoscaler and KAI integration for Ray |
Seminal works
| Year | Work | Contribution |
|---|---|---|
| 2019 | Jeon et al., Analysis of Large-Scale Multi-Tenant GPU Clusters (Microsoft Philly, ATC) | First major study of DL cluster behavior |
| 2018 | Moritz et al., Ray (OSDI) | Distributed framework for AI and RL |
| 2015 | Verma et al., Borg (EuroSys) | Large-scale cluster management; ancestor of Kubernetes |
| 2011 | Ghodsi et al., Dominant Resource Fairness (NSDI) | Fair sharing across multiple resources |
| 2020 | Gandiva, Tiresias, Themis (OSDI/NSDI) | Scheduling designs for deep-learning jobs |
| 2003 | Yoo et al., SLURM | Open HPC workload manager |
Gap in the literature: little published on scheduling mixed simulation, rendering and RL workloads for physical AI.
The platform's job is to give heavy-industry AI teams fair, predictable access to GPUs. Field-level application areas and what each asks of the platform:
| Application area | Workloads that hit the GPU platform | Scheduling implication |
|---|---|---|
| Construction autonomy (excavators, dozers, site prep) | Fleet-data curation; VLA or policy training; simulation and sim-to-real transfer | Large guaranteed training share; preemptible data and simulation classes |
| Mining haulage and drilling autonomy | Perception retraining; log replay; safety-case evaluation runs | Reserved capacity for release gates; reproducible pinned environments |
| In-cab operator assistance | Cloud-side assistants and teacher models; distillation for on-machine models (canvas 7) | Low-latency inference pools; MIG for small models |
| Jobsite digital twins | Reconstruction, rendering, scene generation | RTX pools; batch priority |
A starting queue design (an illustrative pattern, not a recommendation for any particular organization):
| Queue (org level) | Typical users | Guaranteed share | Borrowing | Preemption |
|---|---|---|---|---|
| Release gates | Evaluation (canvas 4) | Reserved windows before releases | No | Never preempted in window |
| Autonomy training | Construction and mining autonomy teams | Largest guaranteed share | Yes | Can reclaim borrowed capacity |
| Simulation and RL | Simulation team (canvas 2), RL research | RTX and datacenter pools | Yes | Preemptible when borrowing |
| Data flywheel | Curation, labeling (canvas 1) | Small guaranteed share | Yes, heavily | Fully preemptible |
| Perception and assistants | Perception and operator-assistance teams | Medium | Yes | Standard |
| Enterprise AI | Other internal AI workloads sharing the cluster | Per agreement | Limited | Standard |
| Interactive | Notebooks, debugging | MIG slices | No | Idle timeout |
Shares are placeholders to be set from demand data in phase 1.
Industry connections
NVIDIA: reference hardware designs, Mission Control, KAI Scheduler, GPU Operator.
GPU clouds and hyperscalers: burst and dedicated capacity; for example, CoreWeave documents Kueue support (CoreWeave).
Autonomy developers and equipment makers: consumers of this platform (Komatsu, Rio Tinto, Bedrock Robotics above); their infrastructure choices are mostly undisclosed.
Data-center operators: power, cooling and network for on-prem clusters (conceptual constraint only here).
Future applications (speculative): a site-level digital twin of the AI data center itself for capacity planning, in line with NVIDIA-partner reference designs (Data Center Frontier, above); selling spare off-peak capacity is a possible but untested idea.
Technology integration
Compute. Dense GPU racks need liquid cooling and high power per rack; NVIDIA documents that a single DGX GB200 rack power shelf delivers up to 33 kW, with eight shelves per rack (NVIDIA). Treated here as a workload requirement, not a design.
Network. Multi-node training needs a high-bandwidth fabric (InfiniBand at 400–800 Gb/s or Spectrum-X Ethernet) and a separate storage network; the requirement is what matters for scheduling.
Storage. High-throughput parallel file system or object store with caching for video and sensor data; checkpoint storage close to GPUs.
Data infrastructure. Data locality matters: move compute to the data lake (canvas 1) or pay egress.
Ethical considerations
Energy and emissions. Idle GPUs waste energy and money; utilization targets are also sustainability targets.
Fair access. Small teams and research projects should not be crowded out by large programs; transparent quotas.
Security and isolation. Shared clusters hold confidential customer and machine data; tenant isolation and access control matter.
Regulatory and policy considerations
| Area | What it means for the platform |
|---|---|
| Data residency (GDPR, national rules, customer contracts) | Some machine and site data must be processed in specific regions; placement rules by data classification |
| Export controls on advanced GPUs | Location and access rules for accelerator hardware and cloud regions |
| EU Energy Efficiency Directive (data-center reporting) | Large data centers in the EU report energy and efficiency metrics |
| Cybersecurity (NIS2 in the EU; customer audits) | Security controls and incident reporting for critical infrastructure |
| EU Machinery Regulation and AI evidence (canvases 4, 5) | Training and evaluation runs must be reproducible: pinned containers, drivers and configs |
Implication: treat the platform blueprint itself as a versioned, audited artifact, because reproducibility of safety-relevant training depends on it.
Build in four phases. Put every GPU behind a queue before adding sophistication; most utilization gains are expected to come from that step (practitioner view, to be confirmed with baselines).
Phases are sequenced, not dated. Phase 1 is highlighted because queues and quotas are expected to deliver most of the utilization gain.
Required resources
Teams: platform engineers (Kubernetes, GPU), HPC and network engineers, SRE, FinOps partner (canvas 8).
Infrastructure: GPU clusters on-prem and in the cloud; fabric; storage; GitOps tooling; monitoring (DCGM, Prometheus).
Partners: hardware vendors, GPU clouds and hyperscalers, data-center operations.
Risk assessment (likelihood and impact ratings are qualitative estimates)
| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| Teams keep private, idle GPU pools | High | High | Central queues with guaranteed quotas so teams don't need to hoard |
| Large training jobs starve | Medium | High | Gang scheduling, reservations, priority for flagship runs |
| Preemption loses work | Medium | Medium | Mandatory checkpointing for preemptible classes |
| Cloud burst costs overrun | Medium | Medium | Budgets per queue; alerts; reserved capacity for steady demand |
| Platform drift between clusters | Medium | Medium | One versioned blueprint deployed by GitOps |
| Network or storage becomes the bottleneck | Medium | High | Topology-aware placement; storage benchmarks before scale-up |
Success metrics
| Metric | What it shows |
|---|---|
| GPU utilization: allocated and actually busy (%) | Efficiency |
| Queue wait time by workload class (hours) | Fairness and capacity |
| Goodput of multi-node training (%) | Reliability at scale |
| Share of GPUs behind a managed queue (%) | Platform adoption |
| Time to stand up a new cluster from code (days) | IaC maturity |
| Cloud burst spend vs budget | Cost control |
Set targets after phase 1 baselines.
Open questions for practitioners
Do physical-AI workloads share an existing AI cluster, rented GPU-cloud capacity, or a dedicated cluster?
Kubernetes only, or Kubernetes plus Slurm for the largest training runs?
Packaged platform (Run:ai, Rafay, OpenShift AI) or upstream components (Kueue, KAI, KubeRay)?
Which data must stay in which regions, and how does that constrain placement?
How are GPU costs charged back to teams (canvas 8)?
What share of capacity is reserved for release gates?
Follow-up prompts (from the research-area framework)
Elaborate on a Kueue ClusterQueue and cohort design for a multi-team physical-AI organization.
Provide detailed methods for co-scheduling Isaac Lab rollouts and VLA learners.
Analyze the trade-offs of Kubernetes versus Slurm for multi-node VLA training.
Suggest a pilot to measure utilization gains from MIG and DRA for inference and notebooks.
Kubernetes AI infrastructure in 2026 (CloudOptimo)
Advancing GPU scheduling and isolation in Kubernetes (Rafay); summary (Pulse)
Kubernetes GPU orchestration in 2026: DRA, KAI, Grove (Spheron); Kubernetes GPU scheduling (Techplained)
Kubernetes binds GPU jobs only when every Pod fits (cloudmagazin)
Kueue; KubeRay with Kueue and gang scheduling (Ray docs); KubeRay v1.5 (Anyscale)
Kueue, DAS and GAIE (arXiv 2602.04900)
Cast AI 2026 State of Kubernetes Optimization; Kueue GPU scheduling cost (Cast AI); TechRadar coverage
Heterogeneity at Hyperscale (OSDI 2026); The Utilization Gap
DGX SuperPOD reference architectures and components (NVIDIA)
AI factory reference designs (Data Center Frontier)
NVIDIA acquires SchedMD (Converge Digest)
Ray with Kueue on a GPU cloud; Kueue on a GPU cloud (vendor documentation)
Komatsu: 10th anniversary of autonomous haulage; Rio Tinto 300th autonomous haul truck (alcircle)
Bedrock Robotics raises $80M (Electrek); Fully automated excavators deployed (BD+C)