This canvas covers how developers of heavy-industry physical AI decide a model is good enough and safe enough to ship: the evaluation harness, replay-based validation, scenario coverage, release gates and the safety case those results feed. It is canvas 4 of 9 in a series on physical-AI platforms for heavy industry (construction, mining, earth-moving autonomy), and follows a research-area exploration framework, with the Emerging (11–12) and Applied (13) modules added.
| Boundary | Owned here | Owned elsewhere |
|---|---|---|
| Test data | Rules for held-out sets and scenario libraries; isolation from training | Building and versioning the data (canvas 1) |
| Simulation | Pass/fail criteria, metrics and statistics on simulated and replay runs | Building simulators, scenes and replays (canvas 2) |
| Training | Release-gating evaluation of candidate checkpoints | Training-time validation and checkpoint selection (canvas 3) |
| Lifecycle | Evaluation results and gate decisions as evidence | Registry, promotion workflow and governance records (canvas 5) |
| Deployment | Shadow-mode and staged-rollout acceptance criteria | Serving, monitoring and rollback mechanics (canvases 5, 7) |
Definition. Evaluation and safety assurance for physical AI is the practice of measuring a model's behavior against defined requirements, and turning those measurements into an argument that the machine is acceptably safe in a stated operating domain. Evaluation produces evidence; assurance organizes it. UL 4600 frames this as a safety case: goals, argumentation and evidence (Koopman).
The evaluation ladder. Each step is cheaper and faster than the next, but less like reality. A model climbs only if it passes the step below.
A model climbs one rung at a time; each rung's results are stored as evidence for the safety case.
Key sub-domains
| Sub-domain | Core question | Typical artifacts |
|---|---|---|
| Requirements and ODD | What must the machine do, and where? | Operational design domain (ODD) spec; safety requirements |
| Evaluation harness | How are models run, scored and compared the same way every time? | Harness code, metric definitions, versioned test suites |
| Offline and open-loop tests | Does the model predict well on held-out data? | Detection, segmentation and action-prediction metrics by slice |
| Closed-loop simulation | Does the policy succeed when its actions change the scene? | Scenario suites in simulation; success and safety metrics |
| Replay-based validation | Would the new model have handled real past events? | Replays of field logs and incidents; regression results |
| World-model evaluation | Can a learned simulator rank policies like reality does? | Correlation studies between world-model and real results |
| Test-site and field validation | Does it work on real machines, in controlled then real conditions? | Proving-ground trials, shadow mode, staged rollout data |
| Safety case | Is the evidence enough to argue acceptable risk? | Claims, arguments, evidence; residual risk; sign-offs |
Debate. Benchmark scores are easy to compare but say little about rare, high-consequence failures. Safety engineers prefer coverage and argument over averages. A platform needs both: averages to track progress, coverage and worst cases to gate release.
Two traditions are converging: safety engineering, which argues from requirements and failure analysis, and machine learning, which measures on benchmarks. Physical AI needs the rigor of the first and the scale of the second.
| Year | Milestone | Why it matters here |
|---|---|---|
| 2026 | Statistically rigorous policy comparison at RSS (Snyder et al.): up to 70% fewer trials; world-model evaluators (WorldGym at ICLR, SC3-Eval) | Evaluation becomes a research field with statistics and learned simulators |
| 2026 | VLA evaluation survey of 582 papers finds benchmarks saturated and claims ahead of evidence (canvas 1) | Benchmark culture alone is not enough |
| 2025 | PolaRiS: real-to-sim evaluation; existing simulation benchmarks correlate poorly with real performance | Replay and reconstruction become evaluation tools |
| 2024 | ISO/PAS 8800 (December): AI safety lifecycle for road vehicles | First standard treating AI and data as safety artifacts |
| 2023 | UL 4600 Edition 3 adds autonomous trucking | Safety-case approach extends to heavy vehicles |
| 2022 | ISO 21448 (SOTIF) published as full standard | Hazards from functional insufficiencies, not faults |
| 2019–2020 | ISO 17757 revised (2019); UL 4600 Edition 1 (2020) | Autonomous earth-moving safety; goal-based safety cases |
| 2016–2018 | RAND "Driving to Safety" (billions of miles needed for statistical proof); scenario-based testing (PEGASUS) | Proof by mileage alone is impossible; scenarios and simulation needed |
| 1990s–2010s | IEC 61508, ISO 13849, ISO 26262; GSN for structured safety arguments | Functional safety and safety-case notation |
Paradigm shifts
From mileage to scenarios. No fleet can drive enough to prove safety statistically; coverage of a scenario space replaces raw distance.
From prescriptive to goal-based standards. UL 4600 asks for a credible argument, not a fixed checklist (ULSE).
From model to data as the safety artifact. ISO/PAS 8800 treats training, validation and test data as controlled safety artifacts (Model Engineers).
From real tests to learned evaluators. World models and real-to-sim reconstructions now rank policies, with measured correlation to reality.
The hard problem is no longer running tests; it is knowing which cheap tests predict real behavior, and how many expensive ones are enough. Research in 2025–2026 attacks both.
Dominant frameworks
Safety case (claims, arguments, evidence). UL 4600 and ISO 5083 together give a claim-based safety case covering the vehicle, infrastructure and processes, with post-deployment monitoring (SAE 2026-01-0527).
AI safety lifecycle. ISO/PAS 8800 adds AI-specific steps on top of functional safety (ISO 26262) and SOTIF (ISO 21448) (UL Solutions). It is written for road vehicles, but its principles increasingly apply to autonomous industrial vehicles (Visage).
Real-to-sim evaluation. PolaRiS turns short videos of real scenes into simulation environments that rank policies like the real world does (RSS 2026).
World models as evaluators. WorldGym runs policies inside an action-conditioned video model and scores them with a VLM; it preserves policy rankings across versions (ICLR 2026). SC3-Eval reports a Pearson correlation of 0.929 with real results across seven VLA policies (arXiv 2606.18610).
Statistical evaluation. Sequential tests stop early when evidence is enough; SureSim combines large-scale simulation with a small number of paired real tests to give confidence intervals (arXiv 2510.04354).
Key research questions
| Question | State of evidence |
|---|---|
| Which offline or simulated metric predicts field performance? | Off-the-shelf simulation benchmarks correlate poorly; reconstructed and world-model evaluators correlate well on manipulation |
| How many real trials are enough? | Studies typically run 10–60 trials (STEP); sequential methods cut that by up to 70% |
| How do we show coverage of rare hazards? | Scenario-based testing with ODD taxonomies; no agreed completeness criterion |
| Can world-model results count as safety evidence? | Useful for ranking; not accepted as primary evidence |
| How do we evaluate a model that changes? | Regression suites and re-qualification; open for continual learning |
| What is "acceptably safe" for autonomous earth-moving? | Set per product in the safety case; no industry-wide numeric target |
Challenges for heavy equipment
Few failures to learn from. Severe incidents are rare, so evidence must come from near misses, simulation and replay.
Mixed sites. People, manned machines and autonomous machines share space; evaluation must cover interactions.
Tests are expensive. A proving-ground trial with a 50-tonne machine costs far more than a manipulation trial on a lab arm.
Standards gap. ISO/PAS 8800 targets road vehicles; ISO 17757 predates learned components; developers must bridge the two.
Debates
Quantitative risk targets vs argument-based assurance without fixed numbers.
Learned evaluators as gates vs learned evaluators as filters only.
End-to-end evaluation of the whole policy vs component tests of perception, planning and control.
Safety assurance is shaped by standards bodies and a few safety researchers; evaluation methods come from robot-learning labs.
| Cluster | Who | Contribution to this canvas |
|---|---|---|
| Standards bodies | ISO/TC 127 (earth-moving machinery, ISO 17757); ISO/TC 22/SC 32 (ISO/PAS 8800, ISO 26262, ISO 21448); UL Standards & Engagement (UL 4600); ISO/IEC JTC 1/SC 42 (AI) | The rules safety cases must satisfy |
| Safety-case researchers | Phil Koopman (Carnegie Mellon; UL 4600 lead author); University of York assurance group (AMLAS for ML assurance); Edge Case Research | Methods for arguing AI safety |
| Policy-evaluation research | Stanford, UC Berkeley, University of Washington, Toyota Research Institute (PolaRiS, WorldGym); Princeton and TRI (Beyond Binary Success); NVIDIA (Cosmos Evaluator, AlpaSim) | Real-to-sim, world-model and statistical evaluation |
| Scenario-based testing | Foretellix, Applied Intuition, ASAM (OpenSCENARIO), PEGASUS successor projects | Scenario languages, coverage metrics |
| Assessors | TÜV SÜD, TÜV Rheinland, UL Solutions, DNV, notified bodies under the EU Machinery Regulation | Third-party assessment of safety cases |
| Industry actors | Komatsu, Hitachi, Epiroc, Sandvik, Liebherr; Bedrock Robotics; Waymo and Aurora (published safety-case material) | Precedent for autonomous-machine safety cases |
Inside heavy-equipment OEMs and autonomy developers (public evidence). Public evidence on how these actors assure autonomy is thin and uneven, and mostly company-reported:
Komatsu. Reports commissioning its 1,000th FrontRunner autonomous haul truck in April 2026 and more than 11.5 billion tonnes moved by the system (company figures, as reported by Mining Weekly). That is operating history in a controlled mining domain; it is not a published safety argument.
Bedrock Robotics. Reports moving from supervised testing to operator-free excavation on live US project sites, with the machine stopping when a person or unauthorized object comes too close (company announcement, GlobeNewswire via Technology Magazine). Independent evidence of its evaluation process was not found.
Waymo and Aurora (road vehicles, the closest published precedent). Waymo published its safety methodologies and readiness-determination process (2020). Aurora publishes the top four levels of its Safety Case Framework as a goal-structured argument covering trucks and passenger vehicles. Neither is for earth-moving machinery.
Other OEMs and miners. Hitachi, Epiroc, Sandvik and Liebherr, and mine-operator autonomy programs, are active in autonomy; this canvas found no public safety-case or evaluation-pipeline documents from them.
Debate. Should heavy-equipment autonomy developers publish safety-case summaries, as Waymo and Aurora have? Openness builds trust with customers, assessors and regulators and lets the field converge on practice; it also exposes design detail and can invite comparisons on incomplete evidence.
Methods group by rung of the evaluation ladder, plus the statistics and assurance methods that span all rungs.
| Rung | Default method | Research frontier |
|---|---|---|
| Offline tests | Held-out sets sliced by ODD factor (terrain, light, weather, machine); precision/recall for safety classes | Slice discovery: automatically finding where models fail |
| Closed-loop simulation | Scenario suites in Isaac Sim or AGX; success, collision and near-miss metrics | Adversarial scenario search for failures; coverage-guided generation |
| Replay-based validation | Re-run recorded field events with the new model; compare decisions to the old model and to the operator | Closed-loop replay on reconstructed scenes (PolaRiS); generative continuation past the log |
| World-model evaluation | — (new) | Action-conditioned video models as evaluators (WorldEval, WorldGym, SC3-Eval) |
| Test-site trials | Fixed protocol, fixed trial counts per condition | Sequential, anytime-valid tests that stop early (RSS 2026) |
| Shadow mode | New model runs in parallel on fielded machines without control; disagreements logged | Automatic triage of disagreements into the flywheel (canvas 1) |
| Staged rollout | Limited sites, then broader release, with monitoring and rollback | Statistical release criteria tied to field monitoring |
Metrics that matter for machines
Safety: missed detections of people in protected zones; minimum separation; emergency stops; interventions per hour.
Task: success rate; cycle time; bucket fill; grade accuracy; partial progress, which separates policies faster than binary success (Snyder et al.).
Robustness: performance by ODD slice; degradation under dust, night, rain and sensor faults.
Calibration: whether confidence scores match observed error rates, so safety monitors can rely on them.
Assurance methods
Hazard analysis (HARA, STPA) linked to test cases.
Structured arguments in GSN, with ML-specific patterns such as AMLAS.
Traceability: requirement → scenario → test run → result → claim.
Runtime monitors (out-of-distribution detection, ODD checks) as part of the argument, not just the model.
Statistics
Confidence intervals on success and failure rates, not point estimates.
Paired comparisons between candidate and current model on the same scenarios.
Bias correction of large simulation results with small real-world samples (SureSim).
Evaluation is where every other canvas is judged. Each interface needs a named contract.
| Connects to | What crosses the boundary | Contract to define |
|---|---|---|
| Canvas 1: data flywheel | Held-out sets, scenario tags, failures found in evaluation | Test-set isolation; failures filed as data requests |
| Canvas 2: simulation | Scenario libraries, replays, closed-loop runs | Simulator qualification evidence; scenario taxonomy |
| Canvas 3: training | Candidate checkpoints in; pass/fail and failure reports out | Standard handoff; training never sees release test sets |
| Canvas 5: MLOps and governance | Gate decisions and evidence attached to registry entries | No promotion without a passed gate and stored evidence |
| Canvas 6: GPU infrastructure | Large batch evaluation jobs | Evaluation job class with guaranteed capacity before releases |
| Canvas 7: inference | Evaluate the optimized model, not just the training checkpoint | Gate runs on the deployed engine (TensorRT, quantized) |
| Canvas 8: observability | Field metrics that confirm or contradict evaluation | Shared metric definitions between evaluation and monitoring |
Outside fields
Safety and systems engineering: hazard analysis, safety cases, functional safety.
Statistics: experimental design, sequential testing, rare-event estimation.
Human factors: how operators and site staff interact with autonomous machines; acceptance testing with users.
Law and insurance: what evidence supports liability and insurance decisions.
Formal methods: verified runtime monitors and safety envelopes around learned components.
Evaluation is shifting toward cheap learned evaluators calibrated by small amounts of real testing, inside safety cases that are maintained continuously rather than written once.
Emerging trends
Calibrated learned evaluators. World models and reconstructed scenes rank policies; small paired real tests correct their bias.
Sequential, statistically sound testing. Stop when evidence is enough, at a stated confidence.
Living safety cases. UL 4600 and ISO 5083 stress post-deployment monitoring, so field data keeps updating the argument (SAE).
Data as a safety artifact. Following ISO/PAS 8800, test data specifications become part of certification.
Evaluation of reasoning. VLAs that explain their plans can be checked step by step, not only on outcomes.
Research gaps in heavy-industry physical AI
| Gap | Why it is open | Opportunity |
|---|---|---|
| Learned evaluators for earth-moving | Correlation studies cover tabletop manipulation, not digging or hauling | Measure world-model and replay correlation with real test-site results |
| ODD taxonomy for construction and mining | No shared scenario taxonomy for job sites | Define one with ISO/TC 127 or ASAM |
| AI safety standard for off-road machines | ISO/PAS 8800 is for road vehicles; ISO 17757 predates learned components | Contribute to standards work; adapt the 8800 lifecycle by analogy |
| Evidence for people-detection safety | Rare events make statistical proof hard | Combine simulation, replay and test-site trials with explicit statistics |
| Evaluating continual learning | Each update needs re-qualification | Delta-qualification: test only what the update could affect |
Potential breakthroughs
A validated learned evaluator for earth-moving that cuts proving-ground trials by an order of magnitude.
A safety case template for learned autonomy in earth-moving that a notified body accepts under the 2027 Machinery Regulation.
Tools and platforms, by layer
| Layer | Open source / standards | Commercial / vendor |
|---|---|---|
| Evaluation harness | LeRobot eval, robosuite / LIBERO harnesses, pytest-style suites, MLflow evaluate | Weights & Biases; Applied Intuition validation tools |
| Scenario definition | ASAM OpenSCENARIO, OpenODD | Foretellix Foretify |
| Simulation and replay | Isaac Sim, AlpaSim, CARLA; PolaRiS tools (arXiv) | NVIDIA NuRec; Applied Intuition |
| World-model evaluators | WorldGym, WorldEval code; Cosmos Predict + Cosmos Reason as judge | Cosmos Evaluator |
| Statistics | SciPy, statsmodels; sequential-test code from STEP and N-SCORE | — |
| Safety-case tooling | GSN editors; AMLAS templates | Ansys medini analyze; Jama; Polarion |
| Runtime monitoring | OOD detectors; ODD monitors | Edge Case Research tools |
Standards to hold open
ISO 17757:2019 (autonomous earth-moving and mining machine safety).
ISO 13849 and IEC 62061 (functional safety of machinery control).
ISO 21448 (SOTIF); ISO/PAS 8800:2024 (AI safety, road vehicles, 172 pages).
UL 4600 Edition 3; ISO 5083 (automated driving safety).
ISO/IEC TR 5469 (functional safety and AI); ISO/IEC 42001 (AI management).
Benchmarks
Manipulation VLAs: LIBERO, SimplerEnv, RoboTwin, PolaRiS environments.
Driving: nuPlan, Waymo Open Sim Agents, NAVSIM.
Construction: none public; a gap.
Journals and conferences
Safety: SAFECOMP, ISSC, IEEE ISSRE; Reliability Engineering & System Safety.
Robotics: RSS, CoRL, ICRA (evaluation papers increasingly appear here).
Automotive: SAE World Congress (safety-case papers).
Funding sources
Industry: developer-funded safety and autonomy programs.
Public: NIST (AI measurement), NSF (Cyber-Physical Systems), NIOSH (mining safety), EU Horizon (trustworthy AI).
Read UL 4600 and ISO/PAS 8800 summaries for the assurance frame, then PolaRiS, WorldGym and the RSS 2026 statistics paper for evaluation methods.
Recent influential works (2024–2026)
| Year | Work | One-line takeaway |
|---|---|---|
| 2026 | Snyder et al., Beyond Binary Success (RSS) | Sequential, anytime-valid comparisons; up to 70% fewer trials; partial-progress metrics separate policies faster |
| 2026 | SC3-Eval | Video world model as evaluator: Pearson 0.929 with real results across 7 VLAs |
| 2026 | Quevedo et al., WorldGym (ICLR) | World-model rollouts preserve policy rankings |
| 2026 | Mudunuri et al., Safety case following UL 4600 and ISO 5083 (SAE) | How to combine the two frameworks |
| 2025 | Jain et al., PolaRiS (RSS 2026) | Real-to-sim reconstruction gives strong real-world correlation; common sim benchmarks do not |
| 2025 | WorldEval | World models rank DexVLA, π₀ and OpenVLA consistently with real results |
| 2025 | SureSim | Small paired real tests correct large-scale simulation; confidence intervals |
| 2025 | STEP | Sequential testing for policy comparison |
| 2024 | ISO/PAS 8800 | AI safety lifecycle; data as safety artifact |
Seminal works
| Year | Work | Contribution |
|---|---|---|
| 2023 | UL 4600 Edition 3 | Goal-based safety case for autonomous products, incl. trucks |
| 2021 | Hawkins et al., AMLAS (University of York) | Assurance argument patterns for ML components |
| 2019 | ISO 17757 | Safety requirements for autonomous earth-moving and mining machines |
| 2016 | Kalra and Paddock, RAND Driving to Safety | Statistical proof by mileage is infeasible |
| 2017 | Shalev-Shwartz et al., Responsibility-Sensitive Safety | Formal rules for safe behavior around others |
| 2011 | Leveson, Engineering a Safer World (STPA) | System-theoretic hazard analysis |
Literature reviews to keep open: the 2026 VLA evaluation-centric survey (582 papers; canvas 1); Koopman's UL 4600 overviews.
Gap in the literature: no published safety case or validated evaluation pipeline for learned autonomy in construction equipment.
Every model that touches machine behavior needs a gate. The depth of the gate depends on how much the model can affect safety.
| Application | Evaluation emphasis | Gate example |
|---|---|---|
| Person and obstacle detection | Recall on people in protected zones, by ODD slice; latency | No regression on safety-class recall in any slice; replay of all past near misses |
| Autonomous excavation and loading | Task success, cycle time, collision-free operation | Closed-loop sim suite + replay + proving-ground sequential test |
| Dozer and compactor autonomy | Grade accuracy; boundary keeping | Scenario coverage of site boundaries and slopes |
| Mining haulage | Interactions with manned vehicles; fleet-level safety | Fleet-scenario simulation; shadow mode before release |
| Operator-assistant and language-model features | Answer accuracy, safety of advice, hallucination rate | Curated question sets; red-teaming for unsafe instructions |
| Predictive maintenance | Precision and lead time of alerts | Backtests on historical failures |
Industry connections
Notified bodies and assessors (TÜV, UL Solutions) for EU Machinery Regulation conformity from January 2027.
NVIDIA: Cosmos Evaluator, AlpaSim and NuRec for simulated and replay evaluation.
Site owners and operators: site acceptance testing and shared incident data.
Standards bodies: ISO/TC 127 for ISO 17757 updates.
Future applications: a customer-facing autonomy readiness report per site, generated from the same evaluation evidence; insurer-ready safety evidence packages.
Technology integration
One harness, many rungs. The same harness runs offline sets, simulation suites, replays and world-model evaluations, so results are comparable and traceable.
Evaluate what ships. Gates run on the optimized engine (TensorRT, quantized) on target-like hardware, because optimization can change behavior (canvas 7).
Compute. Large replay and simulation suites before a release need reserved GPU capacity (canvas 6).
Evidence store. Every gate run stores inputs, versions, results and sign-offs, linked to the registry entry (canvas 5).
Ethical considerations
Who bears residual risk. Workers on mixed sites carry the risk of autonomy failures; acceptance criteria should involve safety representatives, not only engineers.
Metric gaming. Teams under schedule pressure may tune to the gate; keep some test sets hidden from model developers.
Fairness across sites and people. Detection performance must hold across clothing, body types, postures and regions; report by slice.
Honest reporting. Publish failures and limitations in model cards and safety cases, not just pass rates.
Regulatory requirements
| Rule | What it means for evaluation |
|---|---|
| EU Machinery Regulation 2023/1230 (from 20 January 2027) | AI-based safety functions need notified-body assessment; the technical file must contain the test evidence |
| EU AI Act as amended by Regulation (EU) 2026/1744 (in force 27 July 2026) | Machinery is no longer in the Act's direct high-risk scope; equivalent rules are expected through Machinery Regulation delegated acts. Where high-risk duties apply (from 2 Dec 2027 / 2 Aug 2028): testing against defined metrics, logging, accuracy and robustness, post-market monitoring |
| ISO 17757:2019 | Safety requirements and verification for autonomous earth-moving and mining systems |
| ISO 13849 / IEC 62061 | Safety-related control functions must meet performance levels; learned components need a safe architecture around them |
| ISO/PAS 8800 (road vehicles, used as reference) | Data and model verification within an AI safety lifecycle |
| OSHA and MSHA (US) | Site safety duties of employers and mine operators using autonomous machines |
Implication: the evaluation platform is part of the certification path. Its own tools, simulators and world models may need qualification as tools under functional-safety rules.
Build the harness and regression gates first; add learned evaluators only after their correlation with real tests is measured.
Phases are sequenced, not dated. Phase 1 is highlighted because no learned model should reach a machine without passing a stored, repeatable gate.
Required resources
Teams: evaluation engineers; safety engineers (functional safety, SOTIF); statisticians; test-site operations; a release board with authority to block.
Infrastructure: evaluation harness; scenario and test-set repositories with access controls; reserved GPU capacity; evidence store linked to the registry.
Partners: assessors and notified bodies; NVIDIA for replay and world-model tools; standards committees.
Risk assessment
| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| Gates pass models that fail in the field | Medium | High | Replay every past incident; shadow mode; staged rollout with monitoring |
| Test sets leak into training | Medium | High | Access-controlled sets; hash checks against training manifests (canvas 1) |
| Learned evaluators trusted before validated | Medium | High | Use as filters until correlation with real test sites is measured |
| Proving-ground capacity becomes the bottleneck | High | Medium | Sequential tests; better simulation filtering before real trials |
| Safety case not accepted by assessors | Medium | High | Engage notified bodies early; follow UL 4600 and ISO 17757 structure |
| Evaluation slows release cadence too much | Medium | Medium | Risk-tiered gates: lighter for non-safety models, full for safety functions |
Success metrics
| Metric | What it shows |
|---|---|
| Field incidents not caught by any gate (count, trend) | Gate effectiveness |
| Correlation between simulation or replay results and test-site results | Trust in cheap evaluation |
| Real trials per release decision | Evaluation efficiency |
| Past incidents in the replay regression suite (%) | Coverage of known failures |
| ODD scenario coverage (%) | Completeness |
| Time from candidate checkpoint to gate decision (days) | Speed |
| Releases with complete evidence packages (%) | Audit readiness |
Set targets after phase 1 baselines.
Open questions for practitioners
Who should own the release decision for learned models: a product-safety function, the autonomy program, or a joint board with authority to block?
How can existing functional-safety processes absorb ISO/PAS 8800-style data and model requirements?
How many proving-ground trial-days per release cycle does gated testing need, and how can that demand be reduced?
Which models count as safety-relevant and need the full gate?
Is early third-party assessment (before the Machinery Regulation applies in 2027) worth seeking for a pilot safety case?
Should developers publish safety-case summaries for autonomous products?
Follow-up prompts (from the framework)
Elaborate on an ODD and scenario taxonomy for autonomous excavation on construction sites.
Provide detailed methods for validating a world-model evaluator against proving-ground results.
Analyze the challenges of writing a UL 4600-style safety case for a VLA-based excavator.
Suggest a sequential test design for releasing a new person-detection model.
An Overview of Draft UL 4600 (Koopman)
UL 4600 Edition 3 and webinar summary (UL Standards & Engagement)
Developing Safety Case for Automated Driving following UL 4600 & ISO 5083 (SAE 2026-01-0527)
ISO/PAS 8800:2024 (ISO); UL Solutions summary; Model Engineers summary; ISO/PAS 8800 for industrial vehicles (Visage)
PolaRiS (arXiv); RSS 2026 abstract
Beyond Binary Success (RSS 2026); STEP; SureSim
Machinery Regulation 2023/1230: what the Digital Omnibus on AI changes (EUROGIP); Regulation (EU) 2026/1744, Articles 2 and 3
Aurora Safety Case Framework; Waymo's Safety Methodologies and Safety Readiness Determinations (arXiv 2011.00054)
Komatsu 1,000th autonomous haul truck (Mining Weekly); Bedrock Robotics deployments (GlobeNewswire via Technology Magazine)