Simulation and synthetic data fill the gaps that field data can't: rare hazards, unsafe scenarios, new machines before they ship, and repeatable tests. The field in view is physical AI for heavy industry: construction, mining and earth-moving autonomy. This canvas covers generating that data and using simulation in the loop for evaluation. It is canvas 2 of 9 and follows a research-area exploration framework, with the Emerging (11–12) and Applied (13) modules added.
| Boundary | Owned here | Owned elsewhere |
|---|---|---|
| Synthetic data | Generating it: scenes, sensors, randomization, world-model augmentation | Ingesting, versioning and mixing it with real data (canvas 1) |
| Replay | Turning field logs into re-simulatable scenes and variants | Pass/fail rules and scoring of replay runs (canvas 4) |
| Reinforcement learning (RL) | Simulated environments, rewards and physics fidelity | RL algorithms and training loops (canvas 3); GPU scheduling (canvas 6) |
| Digital twins | Machine, sensor and site models used for simulation | Platform-wide reference architecture (canvas 9) |
Definition. Simulation and synthetic data for physical AI means generating sensor data, scenarios and interactive environments in software, so models can be trained and tested where real data is scarce, unsafe or expensive. Two families now work together. Physics simulators such as Isaac Sim model geometry, dynamics and sensors. World foundation models such as Cosmos generate or restyle realistic video. NVIDIA's own workflow chains them: generate in Isaac Sim, then augment with Cosmos to add visual variation and narrow the sim-to-real gap (NVIDIA).
Simulation-in-the-loop means simulation sits inside the data and evaluation pipeline, not beside it: field logs seed scenes, scenes produce data and test cases, and results feed back to the data flywheel.
Field logs and engineering assets enter at the top; the simulator is the shared core; synthetic data returns to canvas 1 and closed-loop runs feed evaluation and RL training.
Key sub-domains
| Sub-domain | Core question | Typical artifacts |
|---|---|---|
| Digital twins and assets | Are machines, sensors and sites modeled faithfully? | USD assets, SimReady models, calibrated sensor models |
| Physics fidelity | Do contact, soil and hydraulics behave like reality? | Physics parameters, terramechanics models, system ID results |
| Sensor simulation | Do camera, LiDAR and radar outputs match real sensors? | Sensor RTX models, noise profiles, dust and lighting effects |
| Scenario generation | Which situations do we need, and how many variants? | Scenario libraries, parameter sweeps, adversarial cases |
| Domain randomization | How much variation makes models robust? | Randomization configs (textures, lighting, physics) |
| World-model augmentation | Can we restyle real or simulated video realistically? | Cosmos Transfer and Predict runs, prompts, control maps |
| Real-to-sim reconstruction | Can a field log become a re-simulatable scene? | Neural reconstructions (NuRec, Gaussian splats), replay scenes |
| Simulated RL environments | Can policies learn safely at scale in simulation? | Isaac Lab environments, rewards, curricula |
| Synthetic-data validation | Is the generated data accurate and useful? | Fidelity metrics, physical-plausibility checks, downstream gains |
Debate. Physics-first teams trust simulators they can inspect. Generative-first teams argue world models capture realism no hand-built scene can. Most production stacks now use both, and the open question is where each one's output can be trusted.
The field moved from hand-built simulators made robust by randomization, to GPU-parallel simulation for reinforcement learning, to generative world models that can restyle or invent scenes. Each wave widened what simulation can cover; none removed the need for real data.
| Year | Milestone | Why it matters here |
|---|---|---|
| 2026 | Waymo World Model on DeepMind's Genie 3 (February): camera and LiDAR output, scenarios the fleet never saw | Generative simulation used for safety testing at production scale |
| 2026 | NVIDIA Physical AI Data Factory Blueprint; Isaac Sim technical paper (arXiv 2606.03551) | Curation, augmentation and evaluation packaged as one reference workflow |
| 2025 | Cosmos Predict 2.5 and Transfer 2.5 (arXiv 2511.00062); Isaac Sim 5.0 open-sourced under Apache 2.0 (release notes) | World models enter the synthetic-data pipeline; the main simulator becomes open |
| 2025 | Cosmos world foundation models first released (CES) | Video generation aimed at physical AI, not media |
| 2021–2023 | Isaac Gym, then Orbit and Isaac Lab: thousands of RL environments in parallel on one GPU | RL training time drops from weeks to hours |
| 2020–2023 | Neural rendering: NeRF (2020), 3D Gaussian splatting (2023) | Real logs become re-renderable scenes (real-to-sim) |
| 2018–2019 | OpenAI Dactyl: robot hand trained fully in simulation, then Rubik's cube with automatic domain randomization (Borgi) | Proof that heavy randomization can cross the reality gap |
| 2017 | Tobin et al., domain randomization; CARLA and AirSim simulators | Randomize rendering so the real world looks like one more variation |
| 2000s–2010s | Physics engines (ODE, Bullet, MuJoCo, PhysX); Gazebo; vendor machine simulators | Simulation used mainly for control design and operator training |
Paradigm shifts
From fidelity to variety. Domain randomization showed a rough simulator with wide variation can beat a precise but narrow one.
From CPU to GPU-parallel. Thousands of environments per GPU made simulated RL practical.
From built to reconstructed. Neural reconstruction turns field logs into scenes instead of modeling them by hand.
From rendering to generation. World models generate or restyle sensor data from prompts and control maps. Waymo notes they can still hallucinate and are early-stage (Ground News summary).
The state of the art is a hybrid stack: physics simulation for geometry, dynamics and ground truth; neural reconstruction for real scenes; world models for visual variety. The hardest open problem is not making data look real but proving it helps before spending the training run.
Dominant frameworks
Physics simulation + generative augmentation. Isaac Sim produces scenes with perfect labels; Cosmos Transfer adds weather, lighting and terrain variation across cameras (NVIDIA blog).
Real-to-sim-to-real. Field logs become 3D Gaussian scenes for closed-loop testing. NVIDIA's Instant NuRec builds one in about 1.5 seconds in a single forward pass (NVIDIA Research), and AlpaSim runs closed-loop tests on them (Radiance Fields).
Generative world simulators. Waymo's Genie 3-based model produces camera and LiDAR for situations the fleet never observed (Waymo).
GPU-parallel RL environments. Isaac Lab runs vectorized environments so policies train in parallel on GPUs (arXiv 2606.03551).
Key research questions
| Question | State of evidence |
|---|---|
| Can we predict whether synthetic data will help, before training on it? | New metrics such as SADGE (2026) find appearance and geometry together predict downstream value; neither alone does |
| Is pixel realism the right target? | PSNR, SSIM, LPIPS and FID measure similarity, not usefulness (arXiv 2510.10203) |
| Can world models be trusted as evaluators? | Promising for rare scenarios; hallucination and short stable horizons limit use as safety evidence |
| How do we simulate soil, rock and hydraulics fast enough for RL? | Particle methods are accurate but slow; coarse-to-fine distillation cuts sampling time to under 1/7 (arXiv 2412.07477) |
| How much randomization is enough? | Still empirical; too little overfits the simulator, too much makes tasks unlearnable |
| Does a generated label stay correct after restyling? | Open; users report geometry drift in Cosmos Transfer outputs (NVIDIA forums) |
Challenges and limitations
Terramechanics. Earth-moving involves contact with deformable, granular material that most robotics simulators don't model well.
Sensor effects. Dust, rain, mud on lenses and vibration matter on job sites and are rarely calibrated against real sensors.
Asset cost. Accurate machine models (hydraulics, linkages, weight) and site models take specialist effort.
Compute cost. World-model generation and neural rendering are GPU-heavy; costs must be weighed against real collection.
Validation. There is no agreed way to show a simulator is good enough to count as safety evidence.
Debates
Physics simulators vs learned world models as the core simulator.
Sim-first training with real fine-tuning vs real-first training with sim for coverage.
Simulation as evidence (regulators may accept it) vs simulation as only a filter before real testing.
NVIDIA dominates the general-purpose stack; a small group of specialists owns heavy-machine physics. The groups to track:
| Cluster | Who | Contribution to this canvas |
|---|---|---|
| Simulation platforms | NVIDIA (Isaac Sim, Isaac Lab, Omniverse, Replicator, NuRec, AlpaSim) | Open-source simulator, synthetic-data writers, neural reconstruction; Isaac Sim 5.0 is Apache 2.0 (release notes) |
| World models | NVIDIA Cosmos (Predict and Transfer 2.5); Google DeepMind Genie 3 and the Waymo World Model; Wayve GAIA | Generated and restyled sensor data; generative closed-loop simulation |
| Heavy-machine physics | Algoryx AGX Dynamics (deformable terrain, AGX Sensor LiDAR); Vortex Studio (CM Labs); Chrono (University of Wisconsin) | Soil, tracks, hydraulics and earth-moving simulation used by machine makers |
| Scenario and AV tooling | Applied Intuition, Foretellix, CARLA | Scenario languages, coverage-driven testing, off-road simulation |
| Academic groups | ETH Zurich RSL (Hutter: legged and excavator RL); UC Berkeley (Abbeel: domain randomization); Stanford, CMU, University of Toronto (Fidler, also NVIDIA); Tampere University (excavator rock capture); Umeå University (AGX-based lunar construction) | Sim-to-real methods; excavation RL; terramechanics |
| Industry peers | Komatsu (with Algoryx on wheel loading), Volvo CE, Built Robotics, Bedrock Robotics | Applied autonomy programs; how much simulation each uses is mostly not public |
Inside heavy-equipment OEMs and autonomy developers (public evidence). Autonomy is commercial in surface-mining haulage and early in construction. Komatsu commissioned its 1,000th ultra-class autonomous haul truck in April 2026 and reports more than 11.5 billion tonnes moved by FrontRunner customers (company-reported; Komatsu). Rio Tinto took delivery of its 300th Komatsu autonomous haul truck in the Pilbara, reported as about 80 per cent of daily production capacity across ten sites (alcircle). On the simulation side, a Komatsu wheel-loading autonomy thesis combined real-world data with AGX Dynamics (Algoryx), and Volvo CE describes using a digital twin of a quarry site to analyse process changes (Agg Business). Volvo Autonomous Solutions and Holcim piloted electric autonomous haulers in a Swiss limestone quarry from 2021 (OEM Off-Highway). Construction-autonomy developers such as Built Robotics (retrofit kits that turn excavators into autonomous trenching machines) and Bedrock Robotics (autonomous excavators, funded in 2026) are visible in the market, but the sources reviewed do not describe their simulation pipelines. Outside the sector, Waymo and NVIDIA publish the most detail on generative and physics-based simulation workflows (above). Public evidence of heavy-industry-specific synthetic-data validation remains thin.
Note: a 2025 excavator RL study from Tampere University (Automation in Construction) trained a PPO policy in AGX Dynamics on a large excavator model that the paper gives as 65,960 kg (arXiv 2510.04168), an example of third-party research on machine models of this size.
Methods group by pipeline stage. Each has a proven default and a frontier.
| Stage | Default method | Research frontier |
|---|---|---|
| Build assets | CAD to USD conversion; SimReady assets; hand-tuned machine models | Automatic asset generation from scans; learned articulation and mass properties |
| Calibrate physics | System identification against test-rig data (hydraulic response, track slip) | Differentiable simulation to fit parameters from logs |
| Model terrain | Heightfields, particles (DEM), hybrid deformable terrain (AGX Terrain) | Coarse-to-fine policy distillation to cut particle-simulation cost (PRPD); learned soil surrogates |
| Simulate sensors | Ray-traced camera and LiDAR (Sensor RTX, AGX Sensor) with noise models | Dust, rain and lens-contamination models validated against real sensors |
| Generate scenarios | Parameter sweeps; scenario description languages; Replicator randomization | Adversarial and coverage-driven scenario search; LLM-written scenarios |
| Augment with world models | Cosmos Transfer from depth, segmentation or edge maps; weather and lighting prompts | Label-preserving augmentation; checking that geometry did not drift |
| Reconstruct real scenes | Per-scene Gaussian splatting (NuRec, 3DGUT) | Feed-forward reconstruction in seconds (Instant NuRec); editable, decomposed scenes |
| Replay and vary | Open-loop log replay; closed-loop replay with the new policy in control | Generative continuation past the end of the log (Waymo World Model) |
| Train in simulation (RL) | PPO in Isaac Lab or AGX; domain randomization; curricula | Sim-trained foundation policies; real-world fine-tuning with safety constraints |
| Validate synthetic data | Train-on-synthetic, test-on-real; FID and similar image metrics | Predictive utility metrics before training (SADGE) |
Data collection to support simulation
System ID campaigns: instrumented machines on test sites, recording actuator commands, pressures and motion.
Soil characterization: cone penetrometer and shear tests to set terrain parameters per site type.
Sensor characterization: paired real and simulated captures in controlled dust, light and weather.
Site capture: drone photogrammetry and LiDAR scans for site twins.
Analysis techniques
Sim-to-real gap measurement: the same policy or detector scored in simulation and on real test sites.
Ablations: which randomization or augmentation actually moves real-world performance.
Coverage analysis: generated scenarios mapped against the operational design domain.
Physics plausibility checks: automated rejection of generated clips that violate basic physics (Cosmos Evaluator).
Simulation sits between the data flywheel and model evaluation. Each interface needs a named contract.
| Connects to | What crosses the boundary | Contract to define |
|---|---|---|
| Canvas 1: data flywheel | Field logs and mined failures in; synthetic data out | Provenance tags (generator, version, seed, prompt); real-to-synthetic mixing rules |
| Canvas 3: training and post-training | RL environments; synthetic demonstrations for VLA training | Environment API (observations, actions, rewards); versioned environment releases |
| Canvas 4: evaluation and safety | Replay scenes, scenario libraries, closed-loop runs | Scenario taxonomy; simulator qualification evidence |
| Canvas 5: MLOps and governance | Simulator and asset versions recorded with every model | Simulator version as part of model lineage |
| Canvas 6: GPU infrastructure | Rendering, world-model and RL jobs on the shared cluster | Job classes for render, generate and RL; mixed graphics and compute GPUs |
| Canvas 7: inference | Policies run in simulation with the same optimized runtime as on the machine | Hardware-in-the-loop and software-in-the-loop interfaces |
| Canvas 8: observability and cost | Cost per synthetic hour; simulation farm utilization | Cost model comparing synthetic and real data |
| Canvas 9: reference architecture | Digital twins shared across engineering and operations | OpenUSD as the shared scene format |
Outside fields
Geotechnical engineering and terramechanics: soil models and parameters.
Hydraulics and machine dynamics: actuator models from manufacturers' engineering data.
Computer graphics: rendering, neural reconstruction, Gaussian splatting.
Systems and safety engineering: when simulation counts as verification evidence (ISO 21448 SOTIF, ISO 17757).
Operator training: training simulators can share machine assets and terrain physics with autonomy simulation.
Surveying and BIM: design surfaces and site models as simulation inputs.
The next step is simulation that is built from field data automatically and generates what the fleet hasn't seen. For heavy equipment, the bottleneck stays physical: soil, rock and hydraulics.
Emerging trends
Instant real-to-sim. Feed-forward reconstruction turns any field log into a test scene in seconds, so every incident can be replayed.
Generative closed-loop simulation. World models continue a scene past the end of a log and react to the policy, as in the Waymo World Model.
Action-conditioned world models. Models such as Cosmos Predict 2.5 generate video conditioned on robot actions, a step toward learned simulators for policy evaluation (arXiv 2511.00062).
Hybrid physics + neural. Physics for contact and soil; neural rendering for appearance. The XSCAVE consortium argues this hybrid is the most promising route for construction and off-road autonomy (Algoryx).
Predictive data-utility metrics. Scoring synthetic data before training on it.
Simulation as regulatory evidence. Pressure to qualify simulators for safety cases.
Research gaps in heavy-industry physical AI
| Gap | Why it is open | Opportunity |
|---|---|---|
| Fast, accurate soil and rock models | Particle methods are accurate but too slow for large-scale RL | Learned soil surrogates trained on test-rig data and particle simulations |
| Hydraulic and machine dynamics fidelity | Public simulators lack calibrated heavy-machine hydraulics | Shared, calibrated machine models released as open or licensed simulation assets |
| Job-site sensor effects | Dust, mud and vibration are rarely validated | Paired real/simulated sensor captures on proving grounds, published as open datasets |
| World models for construction scenes | World models are trained mostly on roads, homes and factories | Post-train open world models on jobsite and mine-site video |
| Multi-machine scenarios | Few simulators handle fleets of interacting machines and people | Site-level simulation for mixed autonomous and manned fleets |
| Simulator qualification | No agreed method to accept simulation as evidence | A qualification process developed with safety engineering and standards bodies |
Potential breakthroughs
A learned simulator for digging that is fast enough for RL and accurate enough to transfer without heavy real-world tuning.
Every field incident replayed and varied automatically within a day.
Tools and platforms, by layer
| Layer | Open source / standards | Commercial / vendor |
|---|---|---|
| Scene format | OpenUSD; URDF/MJCF for robots | NVIDIA Omniverse; SimReady assets |
| Robotics simulation | Isaac Sim, Isaac Lab, MuJoCo, Gazebo, Project Chrono, Genesis | Vortex Studio |
| Heavy-machine physics | Chrono (granular terrain) | AGX Dynamics (with Unity and Unreal plugins), AGX Sensor |
| Synthetic-data generation | Omniverse Replicator, MobilityGen | NVIDIA Physical AI Data Factory Blueprint |
| World models | Cosmos Predict, Transfer and Reason (open weights) | Genie 3 (not open) |
| Neural reconstruction | 3D Gaussian splatting, gsplat; Instant NuRec (GitHub) | NVIDIA NuRec |
| Closed-loop and scenario testing | CARLA, AlpaSim, OpenSCENARIO | Applied Intuition, Foretellix |
| Workflow orchestration | Ray, Kubernetes | NVIDIA OSMO |
Benchmarks
Robot learning: LIBERO, RoboCasa, BEHAVIOR-1K, ManiSkill.
World models: PAI-Bench (used to evaluate Cosmos Predict 2.5).
Off-road and construction: few public simulation benchmarks exist; this is a gap.
Journals and conferences
Robotics: CoRL, RSS, ICRA, IROS; IEEE RA-L.
Graphics and vision: SIGGRAPH, CVPR, ICCV, ECCV.
Field robotics and construction: Journal of Field Robotics; Automation in Construction; ISARC.
Terramechanics: ISTVS conference and Journal of Terramechanics (AGX workshop at ISTVS 2025).
Industry: NVIDIA GTC, SIGGRAPH, CONEXPO-CON/AGG.
Standards
OpenUSD (Alliance for OpenUSD), ASAM OpenSCENARIO and OpenDRIVE.
ISO 21448 (SOTIF), ISO 17757, UL 4600 for how simulation evidence enters a safety case.
Funding sources
Industry: machine-maker and autonomy-developer R&D programs; partnerships with simulation-platform vendors.
Public: NSF (robotics, cyber-physical systems), DOE (mining efficiency), NASA (lunar construction work uses the same terramechanics), Horizon Europe (XSCAVE-style projects).
Read the Cosmos 2.5 paper and the Isaac Sim paper for the platform, then the excavation papers for the heavy-equipment specifics.
Recent influential works (2024–2026)
| Year | Work | One-line takeaway |
|---|---|---|
| 2026 | NVIDIA, Instant NuRec | A driving log becomes a simulatable 3D Gaussian scene in one forward pass |
| 2026 | NVIDIA, Isaac Sim technical paper | Architecture of GPU-accelerated robot simulation with Isaac Lab for RL |
| 2026 | Bartkowiak et al., SADGE | Predicts synthetic-data usefulness from appearance and geometry together |
| 2026 | Waymo, Waymo World Model | Generative multi-sensor simulation for unseen scenarios |
| 2025 | NVIDIA, World Simulation with Video Foundation Models (Cosmos 2.5) | Trained on 200M curated clips; Transfer 2.5 for sim-to-real and real-to-real translation |
| 2025 | Molaei et al., Learning to Capture Rocks using an Excavator (Automation in Construction) | PPO-based RL in AGX Dynamics on a large (~66-tonne) excavator model |
| 2025 | Style-based synthetic-to-real gap profiling | Image metrics like FID don't capture usefulness; profile style and content |
| 2024 | Progressive-Resolution Policy Distillation | Coarse soil simulation first, fine later: sampling time under 1/7 |
| 2024 | Terenzi et al., ETHcavation | Panoptic perception and planning on an autonomous excavator |
Seminal papers
| Year | Work | Contribution |
|---|---|---|
| 2023 | Kerbl et al., 3D Gaussian Splatting | Fast, high-quality neural rendering; basis of NuRec and replay |
| 2021 | Makoviychuk et al., Isaac Gym | GPU-parallel physics for RL at scale |
| 2019 | OpenAI et al., Solving Rubik's Cube with a Robot Hand | Automatic domain randomization |
| 2018 | OpenAI et al., Learning Dexterous In-Hand Manipulation (Dactyl) | Policy trained fully in simulation transferred to a real hand |
| 2017 | Tobin et al., Domain Randomization | Randomize the simulator so reality is one more variant |
| 2017 | Dosovitskiy et al., CARLA | Open simulator for driving research |
| 2020 | Mildenhall et al., NeRF | Neural scene representation from images |
Literature reviews to keep open: Understanding Domain Randomization for Sim-to-real Transfer (theory); the 2026 VLA data-infrastructure survey from canvas 1, which covers simulation-based data engines.
Gap in the literature: almost no published work validates synthetic data or world models for construction and mining scenes.
Autonomy is commercial in mining haulage and emerging in construction (see section 4). Each application below needs simulation for training and validation; the table states what simulation supplies, not what any given company does.
| Application | What simulation supplies | Typical tools |
|---|---|---|
| Perception for autonomy (people, vehicles, obstacles near machines) | Rare and unsafe scenes with perfect labels; weather and dust variants | Isaac Sim + Replicator; Cosmos Transfer |
| Construction autonomy: excavation and loading policies (VLA, RL) | Millions of dig cycles across soil and rock types | AGX Dynamics or Isaac Lab with terrain models |
| Mining haulage autonomy | Haul-road scenarios, interactions with manned vehicles | Scenario tools; site twins |
| In-cab and operator assistance | Test cases for assist functions (collision warning, semi-automated cycles) before field trials | Closed-loop replay; sensor simulation |
| Incident replay and regression testing | Field events rebuilt and varied for every new model | NuRec-style reconstruction; closed-loop replay |
| New-machine development | Autonomy work before a physical prototype exists | Engineering models converted to simulation assets |
| Jobsite and quarry digital twins | Site models for planning and process analysis (Volvo CE example; Swedish research project on mixed autonomous and manned sites) | Omniverse, OpenUSD, operational data |
| Operator training | Shared machine models and terrain physics | Training simulators |
Industry connections
NVIDIA: Isaac Sim, Cosmos, NuRec and the data factory blueprint.
Heavy-machine physics vendors: Algoryx and CM Labs, where terrain fidelity beyond general robotics simulators is required.
Autonomy operators and developers: Komatsu, Volvo Autonomous Solutions, Built Robotics, Bedrock Robotics; mining operators such as Rio Tinto.
Universities: excavation RL and terramechanics groups (ETH, Tampere, Umeå).
Future applications (speculative): site planning that simulates a fleet's productivity before work starts, building on the digital-twin examples above; shared simulation assets and benchmarks for off-road and construction autonomy.
Technology integration
GPU mix. Rendering and sensor simulation need RTX-class GPUs with ray tracing; world-model generation and RL need datacenter GPUs. Isaac Sim and Isaac Lab target RTX PRO 6000 workstations and servers (Neurohive). The cluster in canvas 6 must schedule both kinds.
Simulation farm. Batch scenario runs, replay and generation as queued jobs (Ray, OSMO, Kubernetes) with quotas.
Data needs. Asset library (USD), scenario library, versioned simulator releases, and storage for generated video, which grows fast.
Cost trade-off. Generating one synthetic hour can cost more than collecting a real one for common scenes; simulation pays off on rare, unsafe or not-yet-built cases. Track cost per useful synthetic hour (canvas 8).
Ethical considerations
False confidence. Strong simulation results can hide real-world failures. Never ship on simulation evidence alone for safety functions.
Biased scenarios. Generated scenes reflect what the generator was trained on; underrepresented regions, workers or conditions stay underrepresented.
People in reconstructions. Scenes rebuilt from field logs contain real workers and bystanders; anonymize before reuse.
Synthetic people. Generated humans should cover body types, clothing (including high-visibility gear) and postures common on sites.
Regulatory requirements
| Rule | What it means for simulation |
|---|---|
| EU Machinery Regulation 2023/1230 (from 20 January 2027) | Safety components with self-evolving, machine-learning behaviour that ensure safety functions fall in the third-party-assessment categories (EU-OSHA); simulation evidence should be documented and traceable (design implication, not a legal text) |
| EU AI Act as amended by Regulation (EU) 2026/1744 (in force 27 July 2026) | Machinery moves out of the Act's direct high-risk scope; equivalent requirements are to be brought in through Machinery Regulation delegated acts (Steptoe; sources differ on the delegated-act date). Where high-risk duties apply (from 2 Dec 2027 / 2 Aug 2028), test and validation data, including synthetic, must be relevant, representative and documented |
| ISO 17757:2019 | Safety requirements for autonomous earth-moving and mining systems; verification can include simulation |
| ISO 21448 (SOTIF), UL 4600 | Frameworks for arguing coverage of unknown scenarios, where simulation plays a major part |
Implication: each simulator, asset and world-model version used for validation needs the same version control and lineage as datasets (canvases 1 and 5).
Build in four phases. Qualify the simulator against real data before using it at scale, or the platform will generate a lot of data nobody can trust.
Phases are sequenced, not dated. Phase 1 is highlighted because every later phase relies on a measured sim-to-real gap.
Required resources
Teams: simulation engineers; technical artists for assets; physics and terramechanics specialists; ML engineers for world-model post-training; a safety engineer for simulator qualification.
Infrastructure: RTX rendering nodes, datacenter GPUs for generation and RL, asset and scenario repositories, and a simulation job scheduler.
Partners (examples): platform vendors (NVIDIA: Isaac, Cosmos, NuRec); machine manufacturers for engineering models; specialist terrain-physics vendors (Algoryx, CM Labs).
Risk assessment
| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| Sim-to-real gap larger than expected for digging | High | High | Calibrate on proving-ground data; measure the gap on every policy; keep real fine-tuning in the plan |
| World-model output with wrong geometry or labels | Medium | High | Automated plausibility checks; sampled human review; keep labels from the physics simulator |
| Generation costs outrun value | Medium | Medium | Target only rare or unsafe scenarios; cost per useful synthetic hour reported monthly |
| Asset backlog for many machine models | High | Medium | Convert engineering CAD automatically; prioritize machines with active autonomy programs |
| Simulation evidence rejected in safety cases | Medium | High | Involve safety and certification teams from phase 1; document simulator qualification |
| Lock-in to one simulator | Medium | Medium | OpenUSD assets; simulator-agnostic scenario format (OpenSCENARIO) |
Success metrics
| Metric | What it shows |
|---|---|
| Sim-to-real gap per task (simulated vs real success rate) | Simulator fidelity |
| Real-world gain from adding synthetic data (%) | Usefulness of generation |
| Field incidents replayed in simulation (%) | Coverage of the replay loop |
| Time from field incident to replayable scene (hours) | Speed of real-to-sim |
| Scenario coverage of the ODD (%) | Completeness of the scenario library |
| Cost per useful synthetic hour ($) | Efficiency |
| Simulator versions with qualification records (%) | Readiness to count as safety evidence |
Set targets after phase 1 baselines.
Open questions for practitioners
Is Isaac Sim enough for terrain interaction, or does digging need a specialist engine such as AGX Dynamics?
Which engineering models (hydraulics, linkages) can machine makers release as simulation assets?
Should open world models be post-trained on jobsite video, and who owns and maintains the result?
How should digital-twin and simulation assets be shared between engineering, autonomy and operations teams?
What qualification evidence will safety and certification bodies accept for simulator results?
Which test sites can run paired real/simulated tests for sensor and soil calibration?
Follow-up prompts (from the framework)
Elaborate on a simulator qualification process for autonomous earth-moving machines.
Provide detailed methods for measuring the sim-to-real gap for excavator digging policies.
Analyze the challenges of post-training Cosmos on construction-site video.
Suggest an experiment testing whether Cosmos-augmented synthetic data improves real-world person detection around machines.
Build synthetic data pipelines to train smarter robots with NVIDIA Isaac Sim (NVIDIA)
Isaac Sim 5.0 release notes; Isaac Sim 5.0 overview (Neurohive)
NVIDIA Isaac Sim: Enabling Scalable, GPU-Accelerated Simulation for Robotics (arXiv 2606.03551)
World Simulation with Video Foundation Models for Physical AI (arXiv 2511.00062)
Cosmos Transfer 2.5 and robotics workflows (NVIDIA blog)
Instant NuRec (arXiv 2607.14203); project page
NVIDIA unveils AlpaSim (Radiance Fields)
The Waymo World Model (Waymo); coverage summary (Ground News)
Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World (Tobin et al., 2017); Understanding Domain Randomization
Learning to Capture Rocks using an Excavator; Progressive-Resolution Policy Distillation
Algoryx: heavy off-road autonomy; XSCAVE project; Komatsu wheel-loading thesis; ISTVS 2025 AGX workshop
Komatsu commissions its 1,000th ultra-class autonomous haul truck (Komatsu); Rio Tinto welcomes 300th Komatsu autonomous haulage truck (alcircle)
Embracing the digital quarry (Agg Business); Volvo and Holcim autonomous electric haulers (OEM Off-Highway); Digital twins for electromobility and autonomy in construction machinery (BTH)
Built Robotics funding and Exosystem (Built Robotics); Bedrock Robotics funding (TechCrunch)
Regulation (EU) 2023/1230 (EU-OSHA); EU AI Act amendments enter into force (Steptoe)
NVIDIA SDG forum (user reports on Cosmos Transfer)