This canvas covers how open foundation models are turned into policies and perception models for heavy industrial machines (construction, mining and earth-moving autonomy): supervised fine-tuning, VLA post-training, reinforcement learning (RL), and the multi-GPU training workflows behind them. It is canvas 3 of 9 and follows a research-area exploration framework, with the Emerging (11–12) and Applied (13) modules added.
| Boundary | Owned here | Owned elsewhere |
|---|---|---|
| Data | Data-mix recipes and how datasets are consumed during training | Building and versioning datasets (canvas 1) |
| Simulation | RL algorithms and training loops that use simulated environments | Building the environments and physics (canvas 2) |
| Evaluation | Training-time validation and checkpoint selection | Release-gating evaluation and safety evidence (canvas 4) |
| Model lifecycle | Producing checkpoints with full training metadata | Registry, promotion and governance (canvas 5) |
| Compute | Parallelism strategy, memory and throughput per job | Cluster scheduling, queues and quotas (canvas 6) |
| Deployment | Training for the target latency and hardware (e.g. action chunking for an embedded GPU module) | Quantization, TensorRT and serving (canvas 7) |
Definition. A vision-language-action (VLA) model maps camera images, a language instruction and the machine's state to actions. Most current VLAs pair a pretrained vision-language model (VLM) with an action head. NVIDIA's GR00T N1.6, for example, is a 3-billion-parameter open VLA that combines a Cosmos-family VLM with a 32-layer diffusion transformer action head (NVIDIA).
Post-training adapts such a pretrained model to a specific machine, task and environment. It now has two parts: supervised fine-tuning (SFT) on demonstrations, and RL from rewards, interventions or autonomous practice. The field's working view is that pretrained VLAs still need fine-tuning for real deployment, and imitation alone generalizes poorly outside its data (RL-VLA survey).
Training workflow covers everything around those steps: data loading, parallelism across many GPUs, checkpoints, experiment tracking and reproducibility.
The top row adapts an open model to a heavy machine; the bottom row improves it with real experience and shrinks it for the machine before evaluation and registration.
Key sub-domains
| Sub-domain | Core question | Typical artifacts |
|---|---|---|
| Base model selection | Which pretrained VLA, VLM or perception backbone is the starting point? | Model cards, licence review, baseline scores |
| Embodiment adaptation | How are a machine's joints, hydraulics and cameras mapped into the model? | Action space definition, state normalization, camera config |
| Supervised fine-tuning | How much demonstration data, and which parameters to train? | SFT recipes, LoRA adapters, data-mix ratios |
| RL post-training | How to improve beyond demonstrations? | Reward models, RL configs, intervention logs |
| Perception and world models | How to fine-tune detectors, segmenters and world models for jobsites and mines? | Fine-tuned Cosmos models, perception checkpoints |
| Distributed training | How to train efficiently on many GPUs? | Parallelism configs (FSDP, tensor, pipeline), throughput reports |
| Reproducibility | Can any checkpoint be rebuilt exactly? | Run manifests: code, data version, config, seed, environment |
Debate. One camp builds one large cross-machine foundation model for every machine type. The other fine-tunes smaller specialist models per machine family. The answer depends on whether data from different machines transfers positively, which is still open (canvas 1).
In four years robot learning moved from task-specific policies to open foundation models that are fine-tuned per robot, and then to models that keep improving from deployment through RL.
| Year | Milestone | Why it matters here |
|---|---|---|
| 2026 | Gemini Robotics 2 and On-Device 2 (July): on-device VLA adapts to new embodiments with a few hours of data | Fast embodiment adaptation becomes a product feature |
| 2026 | Physical Intelligence RL tokens (March): precise skills from minutes to hours of real-world RL | Real-world RL becomes practical for fine, precise tasks |
| 2026 | GR00T N1.6 public launch at CES; SimpleVLA-RL at ICLR (proceedings) | Open VLA plus open RL tooling |
| 2025 | π*0.6 with RECAP (November): RL from autonomous rollouts and human corrections more than doubles throughput on the hardest tasks | RL post-training from real deployment |
| 2025 | GR00T N1.6 (December), N1/N1.5 earlier; Gemini Robotics 1.5 (September); OpenVLA-OFT (February) | Open models and a proven fine-tuning recipe |
| 2024 | OpenVLA (7B, open); Octo; π₀ flow-matching VLA | First open VLAs that labs could fine-tune |
| 2023 | RT-2 (VLM backbone emits actions); Open X-Embodiment and RT-X; Diffusion Policy; ACT | VLMs become robot controllers; cross-robot data helps |
| 2022 | RT-1 (Google): transformer policy on 130k demonstrations | Large-scale imitation works on real robots |
| 2017–2021 | Deep RL in simulation (PPO, SAC); sim-to-real locomotion | RL works where simulation is good |
Paradigm shifts
From training per task to fine-tuning a foundation model. Pretrained VLAs carry general visual and language knowledge.
From token actions to continuous action heads. Autoregressive action tokens ran at 3–5 Hz; parallel decoding with action chunks reaches 25 Hz and above (OpenVLA-OFT).
From imitation only to imitation plus RL. As with LLMs, RL after SFT is becoming standard (SimpleVLA-RL).
From one model to planner plus actor. Reasoning models plan and call VLAs to act, as in Gemini Robotics-ER with Gemini Robotics.
The standard recipe in 2026 is pretrained VLM + action expert, fine-tuned on a small machine-specific dataset, then improved with RL. What remains unsettled is how much each stage contributes and how to do RL safely on real machines.
Dominant frameworks
VLM backbone + continuous action head. Diffusion or flow-matching heads predict chunks of actions. GR00T N1.6 predicts state-relative action chunks instead of absolute joint positions (NVIDIA).
Small task-specific post-training. GR00T N1.6 was pretrained for 300K steps; robot experiments then post-trained for 10K–30K steps on small datasets, with co-training on pretraining data to avoid forgetting (NVIDIA).
Optimized fine-tuning recipes. OpenVLA-OFT combines parallel decoding, action chunking, continuous actions and an L1 objective; LIBERO success rose from 76.5% to 97.1% (arXiv 2502.19645).
RL post-training. In simulation (SimpleVLA-RL, RLinf-VLA) and on real robots (RECAP, RL tokens). RECAP learns from demonstrations, autonomous rollouts and human corrections through advantage conditioning (RSS 2026).
Hierarchical systems. A reasoning model plans; the VLA executes, as in the Gemini Robotics-ER plus Gemini Robotics pairing (DeepMind).
Key research questions
| Question | State of evidence |
|---|---|
| How little demonstration data does a new machine need? | Hours, for embodiments close to pretraining data (Gemini Robotics On-Device 2); unknown for hydraulic heavy machines |
| Does RL beat more demonstrations? | SimpleVLA-RL beats SFT in simulation and real tasks under data scarcity (ICLR 2026) |
| Can RL run safely on real machines? | Shown on arms with human supervision; no public evidence found for 20–200-tonne machines |
| Which parameters to tune? | Full fine-tuning, LoRA, frozen backbone with trained action expert all in use; trade-offs depend on data size |
| How to avoid forgetting general skills? | Co-training with pretraining data and regularization; still an active topic |
| Do models transfer across very different bodies? | Positive within arms and humanoids; heavy machines are far from the pretraining mix |
Challenges for heavy equipment
Embodiment gap. Public VLAs are trained on arms and humanoids, not hydraulic booms, tracks and buckets.
Control rates and latency. Machine control loops and safety layers set hard timing limits for on-board inference.
Reward design. Bucket fill, cycle time and grade accuracy are measurable, but safe exploration is not.
Licences. Some open checkpoints are for non-commercial use only, e.g. GR00T N1.6 task models (model card). Check every base model.
Debates
Generalist foundation model vs specialist policies per machine and task.
RL in simulation (safe, scalable, sim-to-real gap) vs RL on real machines (accurate, slow, risky).
End-to-end VLA control vs VLA for planning on top of classical, certified controllers. For safety-critical heavy machines the second looks more likely in the near term (author judgment, not a published finding).
A handful of labs set the VLA agenda; heavy-equipment learning is a small field led by university groups, autonomy start-ups and machine makers in Asia, Europe and North America.
| Cluster | Who | Contribution to this canvas |
|---|---|---|
| Open VLA platforms | NVIDIA GEAR lab (GR00T N1–N1.6; Isaac-GR00T repo); Physical Intelligence (openpi, π₀ to π*0.6); Stanford (OpenVLA, OpenVLA-OFT: Kim, Finn, Liang); Hugging Face LeRobot (SmolVLA) | Open checkpoints and fine-tuning code a platform can adopt |
| Closed frontier VLAs | Google DeepMind (Gemini Robotics 2, ER 2, On-Device 2); Tesla, Figure (in-house) | Benchmarks for what is possible; partner-access models |
| RL post-training | Physical Intelligence (RECAP, RL tokens); Tsinghua/Shanghai AI Lab (SimpleVLA-RL); RLinf team (RLinf-VLA); UC Berkeley (Levine) | Methods for improving VLAs beyond demonstrations |
| Training infrastructure | NVIDIA NeMo (Megatron-Core, NeMo-RL, AutoModel); PyTorch (FSDP2, TorchTitan); Ray; VLA Foundry | Distributed training and RL frameworks |
| Heavy-equipment learning | ETH Zurich RSL (Hutter; HEAP walking excavator); Baidu Research (ExACT, ExT: arXiv 2509.14992); Tampere University (Mattila: IL-ACT, 30-tonne excavator) | The few groups applying imitation, RL and foundation models to excavators |
Inside heavy-equipment OEMs and autonomy developers (public evidence).
Autonomy start-ups. Bedrock Robotics announced in August 2026 that excavators fitted with its system were operating without an operator on live customer sites in Texas and Nevada; press coverage describes a single end-to-end model trained on large amounts of field data, and the safety and productivity claims are company-reported (The Robot Report).
Mining haulage. Komatsu reported commissioning its 1,000th autonomous ultra-class haul truck in April 2026; the figure is company-reported (Skillings Mining Review). Haulage autonomy of this kind is largely perception and planning on fixed routes rather than VLA-style manipulation.
Partnerships. Komatsu has announced an autonomous-mining AI partnership with Applied Intuition (Equipment World). Public detail on training methods in such partnerships is thin.
NVIDIA supplies the most-used open stack for this work (GR00T, Cosmos, NeMo, Isaac Lab) and is the common co-engineering partner for OEMs and start-ups.
No public source reviewed for this canvas documents a production heavy-machine VLA post-trained at scale; most published excavator learning work comes from university groups (see section 9).
Debate. Open models (GR00T, openpi, OpenVLA) give control and on-prem training but lag the closed frontier; closed partner models (Gemini Robotics) may perform better but limit how a machine maker can fine-tune, deploy and certify them.
Methods group by training stage. Each has a default that works today and a frontier.
| Stage | Default method | Research frontier |
|---|---|---|
| Choose a base | Open VLA (GR00T N1.6, π₀/π₀.₅, OpenVLA) or VLM backbone + new action head | Domain pretraining on heavy-machine video and telemetry before task fine-tuning |
| Define the embodiment | Map joints/valves to an action vector; normalize state; state-relative action chunks | Learned embodiment adapters that transfer across machine families |
| Supervised fine-tuning | Full fine-tuning or LoRA on demonstrations; co-train with pretraining data (GR00T N1.6) | Data-efficient recipes; influence-based demo selection (canvas 1) |
| Action generation | Action chunking with diffusion or flow-matching heads; parallel decoding | Real-time chunking for smooth, asynchronous execution on slow hardware |
| RL in simulation | PPO or GRPO-style RL with outcome rewards in Isaac Lab (SimpleVLA-RL) | Massively parallel RL for large VLAs (RLinf-VLA) |
| RL from real experience | Offline RL from logs; advantage-conditioned learning from rollouts and corrections (RECAP) | Online RL with safety constraints and human supervision; RL tokens for precision |
| Distillation | Large teacher to small on-machine student | Distillation that keeps safety-relevant behavior |
| Perception and world models | Fine-tune detectors, segmenters and Cosmos models on jobsite data | Shared backbone between perception and VLA |
Distributed training workflow
Parallelism: data parallel or FSDP2 for models up to a few billion parameters; add tensor and pipeline parallelism for larger ones, keeping tensor parallelism inside a node (NeMo AutoModel guide).
Frameworks: NeMo Megatron-Core for large models; NeMo-RL for RL with Ray-based resource management (NeMo-RL); PyTorch plus the model's own repo for most VLA fine-tuning.
RL systems: rollout workers (simulation or inference), learners and reward services run as separate pools; scheduling them is a canvas 6 problem.
Data loading: sharded, streaming loaders for video and sensor data; GR00T N1.6 added sharded loading (Isaac-GR00T).
Reproducibility: every run records code commit, container, dataset version, config and seed.
Data collection for training
Operator demonstrations via instrumented cabs or teleoperation, paired with machine telemetry.
Autonomous rollouts with human interventions logged as corrections.
Simulation rollouts (canvas 2) for RL and pretraining.
Analysis techniques
Success rate, cycle time and intervention rate per task, in simulation and on test sites.
Ablations of data mix, frozen versus trained layers, chunk size.
Scaling curves: performance versus demonstration hours and RL steps.
Training consumes from canvases 1 and 2 and produces for canvases 4, 5 and 7. Each interface needs a named contract.
| Connects to | What crosses the boundary | Contract to define |
|---|---|---|
| Canvas 1: data flywheel | Versioned datasets and data-mix recipes | Every run pins dataset IDs and manifest hashes |
| Canvas 2: simulation | RL environments and synthetic demonstrations | Environment API and versioned environment releases |
| Canvas 4: evaluation | Candidate checkpoints for release testing | Checkpoint handoff format; training-time metrics vs release metrics |
| Canvas 5: MLOps and governance | Checkpoints with full run metadata | Run manifest schema; registry entry on success |
| Canvas 6: GPU infrastructure | Training and RL jobs, multi-node | Job classes (SFT, RL rollout, RL learner); quotas; preemption rules |
| Canvas 7: inference | Models trained for on-machine latency limits | Target hardware, control rate and chunk size agreed before training |
| Canvas 8: observability and cost | GPU hours, utilization, cost per run | Cost attribution per experiment and team |
Outside fields
Control engineering: how learned policies sit on top of hydraulic controllers and safety layers.
Human factors: operator demonstrations, interventions and trust.
Large language model research: RL post-training methods (GRPO, DPO) transfer from LLMs to VLAs.
Functional safety: learned components inside certified machine control architectures.
The frontier is moving from "fine-tune once" to models that keep learning in deployment. For heavy equipment, the first job is closing the embodiment gap: public VLAs were trained mostly on arms and humanoids, not hydraulic excavators.
Emerging trends
Continual learning from deployment. RECAP-style loops where every shift of autonomous work produces training signal.
Fast embodiment adaptation. Gemini Robotics On-Device 2 adapts to new robot bodies with a few hours of data (DeepMind).
Precision RL with little data. RL tokens learn delicate skills from minutes to hours of practice (Physical Intelligence).
Unified LLM-VLM-VLA training stacks. Frameworks such as VLA Foundry train all three in one codebase (Emergent Mind).
Reasoning plus acting. A reasoning model plans and checks; the VLA acts. Fits jobsite tasks with many steps.
Domain foundation models for construction. Pretraining on diverse excavation demonstrations, then fine-tuning per task, as in ExT (arXiv 2509.14992).
Research gaps in heavy-industry physical AI
| Gap | Why it is open | Opportunity |
|---|---|---|
| VLAs for hydraulic heavy machines | Pretraining mixes have arms and humanoids, not booms and buckets | Domain pretraining on shared excavation demonstrations and telemetry |
| Safe RL on real machines | Exploration on 20–200-tonne machines is dangerous | RL in simulation plus offline RL from logs; constrained online RL on closed test sites |
| Learned policy above certified control | Few published architectures combine VLAs with functional-safety layers | Define a reference pattern: learned planner, certified low-level control |
| Rewards for earth-moving | Productivity, grade quality and machine wear are hard to combine | Reward models trained on operator ratings and telemetry |
| Cross-machine transfer | Unknown whether excavator skills help dozers or loaders | Controlled multi-machine training studies |
| Long-horizon jobsite tasks | Benchmarks cover minutes, not shifts | Hierarchical planning with measurable sub-goals |
Potential breakthroughs (speculative, not forecasts)
A domain foundation policy that adapts to a new machine model with hours of demonstrations instead of months of engineering.
Autonomy that improves regularly from fleet experience, with each update traceable and evaluated.
Models and frameworks
| Layer | Open source | Commercial / partner access |
|---|---|---|
| Open VLAs | GR00T N1.6 (3B), π₀ / π₀.₅ / π₀.₆ via openpi, OpenVLA and OpenVLA-OFT, Octo, SmolVLA | Gemini Robotics 2 and On-Device 2 (early-access partners) |
| VLM and world-model backbones | Cosmos Reason, Qwen-VL, Eagle, PaliGemma | Gemini Robotics-ER 2 (API) |
| Training frameworks | PyTorch FSDP2, TorchTitan, NeMo Megatron-Core and AutoModel, Hugging Face Transformers and LeRobot | NVIDIA AI Enterprise |
| RL frameworks | NeMo-RL, SimpleVLA-RL, RLinf, Isaac Lab RL games / rsl_rl, veRL | — |
| Orchestration | Ray, Kubeflow Training Operator, Slurm | NVIDIA OSMO, Run:ai |
| Experiment tracking | MLflow, Weights & Biases (hosted), Aim | Weights & Biases, Neptune |
| Training compute | DGX systems; GPU clouds | Hyperscale and specialist GPU clouds (e.g. Azure, Nebius, CoreWeave) |
Benchmarks
Manipulation VLAs: LIBERO, SimplerEnv, RoboTwin 2.0, CALVIN, BEHAVIOR-1K.
Excavation: no standard benchmark identified; papers use their own simulators and sites. A shared benchmark is an opportunity.
Journals and conferences
Robotics: CoRL, RSS, ICRA, IROS; IEEE RA-L, T-RO, Science Robotics.
ML: NeurIPS, ICML, ICLR (SimpleVLA-RL appeared at ICLR 2026).
Construction: Automation in Construction; ISARC; ICRA workshops on construction robotics.
Industry: NVIDIA GTC, CES, CONEXPO-CON/AGG.
Funding sources
Public: NSF National Robotics Initiative and Foundational Research in Robotics; DOE; Horizon Europe (construction robotics); Business Finland (Tampere's heavy-machine programs).
Private: venture funding for autonomy start-ups, and OEM and mining-operator R&D budgets.
Read OpenVLA-OFT for fine-tuning practice, π*0.6 for RL from deployment, and ExT and IL-ACT for excavators.
Recent influential works (2025–2026)
| Year | Work | One-line takeaway |
|---|---|---|
| 2026 | Heydari Shahna et al., IL-ACT | Imitation policy plus adaptive Cartesian tracking on a 30-tonne excavator model |
| 2026 | Li et al., SimpleVLA-RL (ICLR) | Outcome-reward RL beats SFT under data scarcity |
| 2026 | RLinf-VLA (RSS) | Unified, efficient system for RL on VLAs |
| 2026 | VLA Foundry | One codebase for LLM, VLM and VLA training; backbone quality matters most |
| 2026 | Physical Intelligence, RL tokens | Precise skills from short real-world RL |
| 2025 | Physical Intelligence, π*0.6 / RECAP | RL from rollouts and corrections doubles throughput on hard tasks |
| 2025 | NVIDIA, GR00T N1.6 | Open 3B VLA; small post-training sets with co-training |
| 2025 | Kim, Finn, Liang, OpenVLA-OFT (RSS) | Fine-tuning recipe: 76.5% to 97.1% on LIBERO; 26x faster action generation |
| 2025 | ExT | Pretraining on diverse excavation demos for task-level generalization |
| 2024 | Chen et al., ExACT | End-to-end imitation learning from LiDAR, camera and joints to excavator valves |
Seminal papers
| Year | Work | Contribution |
|---|---|---|
| 2024 | Kim et al., OpenVLA | First widely used open 7B VLA |
| 2024 | Black et al., π₀ | Flow-matching action expert on a VLM |
| 2023 | Brohan et al., RT-2 | VLM fine-tuned to output robot actions |
| 2023 | Chi et al., Diffusion Policy; Zhao et al., ACT | Action chunking and diffusion for visuomotor control |
| 2022 | Brohan et al., RT-1 | Transformer policy at scale on real robots |
| 2021 | Hu et al., LoRA | Parameter-efficient fine-tuning |
| 2017 | Schulman et al., PPO | Default RL algorithm in simulation |
Literature reviews to keep open: RL of VLA models survey (TechRxiv 2025); VLA post-training and human motor learning; the 2026 VLA data-infrastructure survey (canvas 1).
Gap in the literature: none of the sources reviewed here reports a VLA post-trained and evaluated on a full-size excavator or dozer at production scale.
Models in heavy-industry physical AI fall into three kinds: perception for safety and autonomy, policies for machine control, and language-and-vision models for operator assistance. All three need the same training platform.
| Application | Model type | Training approach |
|---|---|---|
| Person and obstacle detection around machines | Perception (detector, segmenter) | SFT on fleet and synthetic data; frequent retraining from the data flywheel |
| Construction autonomy: excavator digging, loading, trenching | VLA or imitation policy | Demonstrations from skilled operators, then RL in simulation and offline RL from logs |
| Dozer grading, compactor passes | Policy with planner | Imitation plus RL on grade-accuracy rewards |
| Mining haulage and drilling autonomy | Perception and planning | Supervised training on fleet data; scenario-driven retraining |
| In-cab and operator assistance | Speech, VLM and compact LLM | Fine-tuning on manuals, fault codes and service data; runs on embedded GPU modules |
| Jobsite digital twins and understanding | World models (Cosmos post-training) | Fine-tune on jobsite video for prediction and augmentation |
| Predictive maintenance | Time-series models | Supervised training on telemetry and repair records |
Industry connections
NVIDIA: GR00T, Cosmos and NeMo; DGX systems.
GPU clouds: burst and sustained training capacity.
Autonomy developers and OEMs: start-ups such as Bedrock Robotics and mining-haulage vendors such as Komatsu (section 4) are the main consumers of this training stack.
Physical Intelligence, Google DeepMind: possible model sources if open models fall short.
Future applications (speculative): one foundation policy family across excavators, loaders and dozers; operator-assist features (auto-dig, auto-grade) trained with the same stack as full autonomy, so data and models are shared.
Technology integration
Compute scale. Fine-tuning a 3B VLA fits on one 8-GPU node; GR00T N1.6 offers a fine-tuning playbook on a single DGX Station (NVIDIA). Pretraining a domain foundation model or large-scale RL needs multi-node clusters.
RL is a systems problem. Rollouts (simulation or inference), learners and reward models run concurrently. RLinf-VLA allocates GPUs separately for rendering, inference and training (RSS 2026).
Data throughput. Video and multi-sensor training is often I/O-bound; sharded, cached loaders and local NVMe matter as much as GPUs.
Train for the target. Model size, control rate and action chunking must fit the machine's on-board compute budget (canvas 7).
Ethical considerations
Operator data and skills. Training on skilled operators' demonstrations raises consent, credit and workforce questions; agree terms with operators and unions where relevant.
Learning from interventions. Recording human corrections must not become hidden performance monitoring.
Unsafe exploration. RL on real machines should run only on closed test sites with safety layers that the policy cannot override.
Opacity. End-to-end policies are hard to explain after an incident; keep logs that allow replay (canvas 2) and reasoning traces where models produce them.
Regulatory requirements
| Rule | What it means for training |
|---|---|
| EU Machinery Regulation 2023/1230 (from 20 January 2027) | Machines whose safety functions use self-learning behavior need third-party assessment; models that keep learning after sale face the strictest route |
| EU AI Act as amended by Regulation (EU) 2026/1744 (in force 27 July 2026) | Machinery is no longer in the Act's direct high-risk scope; equivalent rules are expected through Machinery Regulation delegated acts. Where high-risk duties apply (from 2 Dec 2027 / 2 Aug 2028): documented training methods, data governance and logging |
| ISO 17757:2019; ISO 13849 | Learned components must sit inside a safety architecture that meets functional-safety requirements |
| Model licences | Some open checkpoints (e.g. GR00T task models) are non-commercial; legal review before production use |
Implication: in the near term, train offline and release frozen, versioned models through evaluation gates (canvas 4). Learning on customer machines after sale is a regulatory step change and needs its own decision.
Build in four phases. Make training reproducible before making it clever: RL and continual learning amplify any untracked change in data or code.
Phases are sequenced, not dated. Phase 1 is highlighted because RL and foundation-scale training only pay off once every run can be reproduced.
Required resources
Teams: ML research engineers (VLA, RL); training-infrastructure engineers; controls engineers for embodiment and safety layers; operators for demonstrations; test-site staff.
Infrastructure: multi-node GPU cluster with fast interconnect; shared storage with high read throughput; experiment tracking; container registry.
Partners: NVIDIA (GR00T, NeMo, Cosmos); GPU cloud for burst capacity; universities for excavation RL.
Risk assessment (likelihood and impact are the author's qualitative estimates)
| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| Open VLAs transfer poorly to hydraulic machines | High | High | Budget for domain-specific pretraining; keep a classical or imitation baseline |
| RL causes unsafe behavior on test machines | Medium | High | RL in simulation first; offline RL from logs; hard safety envelopes on real machines |
| Runs can't be reproduced | Medium | High | Run manifests mandatory; registry rejects checkpoints without them (canvas 5) |
| GPU shortage slows iteration | Medium | Medium | Quotas and priority classes (canvas 6); cloud burst |
| Licence limits on base models | Medium | High | Legal review per base model; prefer permissive licences for production |
| Catastrophic forgetting during post-training | Medium | Medium | Co-training with pretraining data; regression suites (canvas 4) |
Success metrics
| Metric | What it shows |
|---|---|
| Task success rate on test sites, per machine and task | Policy quality |
| Interventions per operating hour | Readiness for autonomy |
| Demonstration hours needed for a new task | Data efficiency |
| Gain from RL over SFT (%) | Value of RL post-training |
| Time from new data to evaluated checkpoint (days) | Iteration speed |
| GPU utilization of training jobs (%) | Infrastructure efficiency |
| Runs with complete manifests (%) | Reproducibility |
Set targets after phase 1 baselines.
Open questions for practitioners
Which base model family (GR00T, openpi, a VLM plus a custom action head) is the default, and which licences allow production use?
Build one cross-machine foundation policy, or specialist policies per machine family?
Where does the learned policy stop and certified machine control begin?
How many operator demonstration hours can be collected per machine type per year, and under what agreement with operators?
Is RL on real machines acceptable on closed test sites, and under which safety envelope?
Should any model keep learning on customer machines after sale, given the 2027 Machinery Regulation?
Follow-up prompts (from the research-area framework)
Elaborate on an action-space design for hydraulic excavators that works with GR00T N1.6 or π₀-style action heads.
Provide detailed methods for RL post-training of a digging policy: simulation, offline RL and supervised real-world RL.
Analyze the challenges of combining a VLA planner with a certified low-level controller.
Suggest an experiment testing whether excavator data improves a wheel-loader policy.
GR00T N1.6 and experiments (NVIDIA Research); GR00T N1.6 fine-tuning playbook (NVIDIA); Isaac-GR00T repository
GR00T N1.6 task model card (licence terms: non-commercial use only)
π*0.6: a VLA That Learns From Experience (arXiv); RSS 2026 version; blog; RL tokens (Physical Intelligence); openpi
SimpleVLA-RL (ICLR 2026); technical report
RLinf-VLA (RSS 2026)
Survey on RL of VLA models; VLA post-training and human motor learning
OpenVLA-OFT (Kim, Finn, Liang)
Gemini Robotics 2 (Google DeepMind)
VLA Foundry (Emergent Mind)
NeMo-RL (repository); NeMo AutoModel distributed training guide
IL-ACT; ExT; ExACT; HEAP walking excavator (ETH Zurich)
Bedrock Robotics operator-free excavator deployments (The Robot Report); Komatsu 1,000 autonomous trucks (Skillings Mining Review); Komatsu and Applied Intuition (Equipment World)