Canvas 8 — Observability, SLOs & GPU Cost (FinOps) for Physical AI

Nov 11, 2026

How to use this canvas

This canvas covers how a heavy-industry physical-AI platform can be seen, promised and paid for. It is part of a series on physical-AI platforms for heavy industry (construction, mining and earth-moving autonomy). It spans three layers: telemetry from GPUs, clusters, models and machines; SLOs that turn that telemetry into commitments; and GPU cost and energy accounting that turns it into unit economics. It is canvas 8 of 9. It follows a research-area exploration framework, with the Emerging (11–12) and Applied (13) modules added.

It takes inputs from Canvas 6 (GPU orchestration) and Canvas 7 (inference serving). It feeds Canvas 9 (reference architecture).

1. Core definition and scope

Definition. Observability, SLOs and GPU cost for physical AI is the discipline of instrumenting the AI factory and the machine fleet so that every GPU-hour, model call and on-machine inference can be explained, held to a stated objective, and priced per unit of useful work.

Three sources share one labelled pipe. The same data then drives three decisions: whether to ship or roll back, how to schedule, and who pays.

Boundaries. In scope: telemetry pipelines, SLO design and error budgets, alerting and incident response for AI workloads, GPU utilization and goodput, cost allocation and chargeback, and energy accounting. Out of scope: how models are evaluated for safety (Canvas 4), how they are versioned and approved (Canvas 5), and how GPUs are scheduled (Canvas 6). This canvas measures those systems; it does not run them.

Key sub-domains

1. GPU and cluster telemetry. Health, saturation and errors from DCGM, NVLink, InfiniBand/Ethernet and power and cooling systems.

2. Training observability and goodput. Step time, checkpoint overhead, failure and restart time, and the share of paid time that becomes training progress.

3. Inference and model observability. Latency percentiles, throughput, queueing, model quality, drift and data-quality signals.

4. Edge and fleet observability. Control-loop latency, thermal throttling and model behavior on Jetson-class machine computers, under intermittent connectivity.

5. SLO engineering. Choosing SLIs, setting targets per consumer tier, burn-rate alerting and error-budget policy.

6. AI FinOps. Cost allocation for shared GPU clusters, unit economics (cost per training run, per million tokens, per machine-hour of autonomy), forecasting and showback/chargeback.

7. Energy and sustainability accounting. Joules per inference or training step, facility PUE and regulatory reporting.

Central debate. Classic GPU "utilization" is a time-busy measure, not a work measure. The field is moving from utilization to goodput and yield: the share of paid capacity that becomes useful output.

2. Historical context and evolution

The field merged three separate lines of work: web-scale SRE, ML monitoring and cloud FinOps. GPU-heavy AI forced them together after 2023.

Paradigm shifts

• From host metrics to work metrics. A busy GPU is not a productive GPU. Goodput, SM activity and tokens per watt replaced "GPU util %".

• From cost reports to unit economics. Teams now ask what one training run or one hour of autonomous haulage costs, not just what the monthly invoice was.

• From cloud-only to edge-plus-cloud. Robot fleets brought observability under network partition and on battery or engine power.

3. Current state of research

Measurement is the weak link. Average enterprise GPU utilization across 23,000+ Kubernetes clusters was about 5% in Cast AI's 2026 report (Cast AI). Even that number overstates useful work, because the common metric only counts time when any kernel runs.

Dominant frameworks

• SRE SLO model. SLIs → SLOs → error budgets → multi-window burn-rate alerts. Applied to inference endpoints and, increasingly, to training platforms.

• Goodput and badput. Google's ML Productivity Goodput splits efficiency into scheduling goodput (resources available when needed), runtime goodput (time spent making progress) and program goodput (how well the code uses the hardware) (MLSys 2026).

• AI factory yield. NVIDIA's operating KPI: useful output per GPU-hour and per megawatt, such as tokens per watt, rather than raw utilization (NVIDIA).

• FinOps Inform → Optimize → Operate. With FOCUS as the common billing schema and unit economics as the target metric.

• OpenTelemetry as the common pipe. Traces, metrics and logs with the gen_ai.* namespace for model calls and agents.

Key research questions

1. Which SLIs capture a physical-AI model's health: latency, action smoothness, intervention rate, or task success?

2. How do you allocate the cost of a shared GPU cluster fairly when jobs are preemptible, fractional (MIG) or idle-reserved?

3. Can anomaly detection on telemetry predict GPU and network failures early enough to checkpoint or drain?

4. What is the right unit of value for an industrial AI service: cost per training run, per validated model, per autonomous machine-hour?

5. How do you observe edge models when the machine is offline for hours, without uploading everything?

6. How do you measure energy per inference consistently across chip, server and facility boundaries?

Main methodological approaches: telemetry-pipeline engineering, statistical SLO analysis, causal and topology-aware root-cause analysis, queueing and capacity models, cost-allocation models, and energy measurement with stated system boundaries.

Challenges and limits

• DCGM_FI_DEV_GPU_UTIL reads 100% when one small kernel runs on a few SMs, so dashboards can hide a mostly idle fleet (Last9).

• Profiling counters (SM active, tensor active) are multiplexed, vary by GPU model and lose accuracy as you add more.

• High-cardinality labels (pod × model × machine × site) make telemetry itself expensive.

• OTel GenAI conventions are still at Development stability, so attribute names can change.

• Energy figures are rarely comparable. Published per-query energy figures were found to be overstated by 4–20× in a 2026 analysis, depending on the boundary chosen (IntuitionLabs).

Debate. Should cost and SLOs be managed per team (chargeback) or per platform (shared budget)? Chargeback drives efficiency but can push teams to under-provision safety-critical validation runs.

4. Key stakeholders and research centers

Practice leads research here: most advances come from operators of large GPU fleets and from standards bodies, with a few academic labs on energy and ML systems.

Influential people

Organizations and groups

• NVIDIA. DCGM, Mission Control, Minerva, Run:ai and the Enterprise Reference Architecture observability guide. The reference stack for most GPU-factory observability.

• Google and Meta. Published fleet efficiency and failure data from very large training runs. Meta's Llama 3 report is a key reference for hardware interruptions during long runs.

• CNCF. OpenTelemetry, Prometheus, OpenCost (Incubating) and llm-d.

• FinOps Foundation (Linux Foundation). FinOps framework, FinOps for AI guidance, FOCUS, State of FinOps report.

• MLCommons. MLPerf Inference and MLPerf Power for comparable performance-per-watt.

• Uptime Institute and The Green Grid. Data-centre resilience tiers and PUE.

• Heavy-equipment OEMs and autonomy developers. Komatsu, Deere, Volvo CE, Hitachi, Built Robotics, Pronto, Teleo and Bedrock Robotics field or develop autonomy and telemetry products. The sources reviewed for this canvas contain no public SLO, goodput or cost-per-machine-hour data from them, which is itself a finding (see Research gaps).

• Mining autonomy operators. Fortescue reported 193 autonomous haul trucks in operation and more than 2 billion tonnes hauled by July 2021 (Global Mining Review). Fleets of this size are the natural setting for the fleet-observability questions in this canvas.

• Robotics observability. Canonical (COS for Robotics on Ubuntu, including Jetson Thor), Formant, and the ROS 2 tracing community (ros2_tracing, ros-opentelemetry).

• Vendors. Datadog, Grafana Labs, Dynatrace, Cast AI, Kubecost (IBM), Finout, Arize, WhyLabs, Evidently.

Collaboration note. Most AI-factory tooling converges on NVIDIA's reference stack and CNCF standards. For an operator buying or integrating tools, OTel and FOCUS compatibility is therefore a sensible procurement requirement (a judgment, not a published rule).

5. Methodological landscape

The core method is a closed loop: instrument, define SLIs, set SLOs, attribute cost, then act on burn rate and unit cost.

Primary methods

1. SLO design. Pick SLIs per consumer (for example p99 time-to-first-action, availability of the training queue, model freshness). Set targets from user tolerance, not from current performance. Use multi-window, multi-burn-rate alerts (for example 2% of a 30-day budget in 1 hour pages; 5% in 6 hours opens a ticket).

2. Goodput accounting. Tag every second of a training job as productive or as a badput bucket: waiting in queue, initialization, checkpoint save and restore, failure and re-run, or slow steps. Google's ml-goodput-measurement library is the reference design (Google Cloud).

3. GPU work profiling. Read DCGM profiling fields instead of GPU_UTIL. NVIDIA's guidance: SM active of 0.8 or higher is necessary but not sufficient for effective use; below 0.5 likely means ineffective use (Last9, quoting DCGM docs).

4. Cost allocation. Allocate by max(request, usage) per workload, as the OpenCost spec does for GPU (OpenCost via DEV). Shared platform cost is split by GPU-seconds or tokens. Report in FOCUS columns.

5. Topology-aware root-cause analysis. Correlate job slowdowns with the exact GPU, NIC, switch and cable through a dependency graph, as in NVIDIA Minerva.

6. Energy measurement. Read GPU power counters with tools such as Zeus. State the boundary: chip, node, rack or facility (with PUE).

Data collection

Analysis techniques: percentile and tail analysis, burn-rate math, change-point detection on step time and latency, anomaly detection on device telemetry, survival models for hardware failure, queueing models for capacity, and cost-per-unit regression against utilization.

6. Interdisciplinary connections

This area sits between platform engineering and finance, and it reaches down to facilities and out to machines in the field.

Cross-disciplinary opportunity. Industrial condition monitoring (fault codes, trend analysis, predictive maintenance) is a mature practice for heavy equipment. The same methods plausibly apply to GPUs and to on-machine AI computers, and reusing them may be cheaper than building a separate AI-only practice (a hypothesis; no published comparison was found).

7. Future directions

The next step is to tie efficiency, reliability and cost into one number per workload, and to let agents act on it.

Emerging trends

• Yield as the top-line KPI. Useful output per GPU-hour and per megawatt (tokens per watt; for physical AI, validated policies or autonomous machine-hours per GPU-hour).

• AIOps agents for the AI factory. Minerva's Sherlock assistant and NVIDIA's AI Factory Operations Agent blueprint expose telemetry through MCP for agent-led diagnosis (NGC).

• Token- and model-level cost. OpenCost 1.121 reports allocation-based and usage-based cost per million tokens, split into prefill and decode (OpenCost).

• FOCUS for AI. AI workloads were the most requested area for FOCUS to expand into next in the State of FinOps 2026 survey (UD).

• Compute-to-facility observability. Power, cooling and leak detection joined to job telemetry, because the gap between them is where many AI outages start (Equinix).

• Energy as an SLO. Joules per action or per training step tracked next to latency, driven by power-capped sites and EU data-centre labels.

Research gaps in heavy-industry physical AI

1. No agreed SLIs for VLA and robot policies in the field (action latency is easy; "behaving well" is not).

2. No public SLO, goodput or cost-per-machine-hour data from heavy-equipment OEMs or autonomy developers was found in the sources reviewed.

3. Little published work on observability under long network partitions for heavy equipment.

4. Fair cost allocation for safety and validation workloads that must run regardless of budget.

5. Goodput definitions for RL and simulation-heavy training, where rollouts and training share GPUs.

6. Low-overhead profiling on edge SoCs such as Jetson Thor without hurting control-loop timing.

Potential breakthroughs

• A physical-AI goodput measure: the share of fleet and factory GPU time that becomes improved field behavior.

• Closed-loop FinOps. Schedulers that read cost and SLO burn in real time and move work across cloud, on-prem and edge.

• Predictive GPU health. Failure prediction that drains and checkpoints before an XID error stops a multi-day run.

8. Resources and infrastructure

The open-source core (OTel, Prometheus, DCGM Exporter, OpenCost) is mature enough to standardize on. NVIDIA and commercial tools sit on top.

Tools and platforms

Standards and specifications: OpenTelemetry semantic conventions; FOCUS 1.3; FinOps Framework; EU Delegated Regulation 2024/1364 (data-centre reporting); ISO/IEC 30134 (data-centre KPIs incl. PUE); ISO/IEC 42001 (AI management system, for monitoring obligations).

Venues: USENIX SREcon, MLSys, OSDI/NSDI/SOSP/EuroSys, SoCC, KubeCon + CloudNativeCon, FinOps X, ROSCon, IEEE CLOUD, ACM e-Energy and NeurIPS Datasets and Benchmarks (energy benchmarks).

Funding: vendor and cloud partner programs; DOE and NSF programs on energy-efficient computing; EU Horizon Europe calls on green and edge computing.

9. Literature analysis

The literature splits into SRE practice books, GPU cluster trace studies, and a newer energy and goodput line. Robot-fleet observability is mostly in practitioner talks, not papers.

Seminal works

Recent influential works

Reviews and reports: Paleyes et al., "Challenges in Deploying Machine Learning" (ACM Computing Surveys, 2022); FinOps Foundation, State of FinOps 2026 (98% of respondents now manage AI spend, up from 63% a year earlier, as summarized by GMI Cloud); NVIDIA Enterprise Reference Architecture observability guide.

Gap. No peer-reviewed study located in this review covers SLOs or cost per unit for physical-AI fleets of heavy machines. Published work here would be new.

10. Practical applications

In practice this canvas becomes a tiered SLO catalog, one cost model, and one set of dashboards shared by platform, infrastructure and product teams.

Proposed SLO catalog (starting points to validate, not commitments)

Field-level applications (illustrative SLIs to validate; proposals, not published standards)

Unit economics to report

• Cost per training run and per validated model (a run only counts when it passes Canvas 4 gates).

• Cost per 1,000 simulation scenarios.

• Cost per million tokens or per million actions served in the cloud.

• AI cost per autonomous machine-hour. Edge compute amortization plus cloud inference plus a share of training, set against the value of that hour.

Current real-world applications (industry)

• GPU cloud providers use goodput and yield dashboards (Mission Control, Minerva) to sell uptime and token output.

• Robot fleet operators run OTel and FluentBit on robots and treat robots as edge nodes in a distributed system, with partition-tolerant observability (SREcon26 Americas, Agility Robotics).

• FinOps teams split shared Kubernetes GPU clusters by GPU-seconds or tokens, starting with showback before chargeback (GMI Cloud).

Future applications (speculative): customer-facing AI service SLAs, cost-aware model selection on each machine, carbon-aware scheduling of non-urgent training, and AI-health fields in machine-health reporting.

11–12. Technology integration, ethics and regulation

Technology integration

• One telemetry pipe. OTel collectors on every GPU node, every service and every machine computer, feeding a shared metrics, logs and traces store. DCGM Exporter runs with pod mapping enabled. Every signal carries the same labels: site, machine serial, model version, team, cost center.

• Edge buffering. Jetson-side collectors store and forward through outages, with priority tiers: safety events first, SLO counters next, debug traces only on request. This reuses the Canvas 1 uploader.

• Compute needs. Telemetry should cost no more than a few percent of the GPU budget. Use sampling, recording rules and short retention for high-cardinality series.

• Cost data. FOCUS exports from each cloud, plus on-prem GPU amortization and power, joined to OpenCost allocations in the lakehouse.

• Integration with Canvases 5–7. Model registry IDs (Canvas 5) appear as labels; Kueue/KAI queue metrics (Canvas 6) feed scheduling goodput; Triton/NIM and Thor engine metrics (Canvas 7) feed inference SLIs.

Ethical considerations

• Operator monitoring. Machine telemetry can expose how an individual operator works. Aggregate by default, limit access, and agree on use with works councils where they exist.

• Cost pressure vs safety. Chargeback must not make teams skip validation runs. Safety and evaluation workloads should get a protected budget.

• Honest energy claims. Report energy with its boundary (chip, node, facility) to avoid greenwashing.

Privacy and data rights

• Camera and audio traces from machines may contain personal data (GDPR in the EU). Redact at the edge and keep debug traces short-lived.

• The EU Data Act gives users of connected products rights to access the data those products generate. Telemetry design should assume customers can request their machine's AI telemetry.

Regulatory requirements (as of Nov 2026)

13. Implementation framework

Roll out in four phases over about 18 months: instrument first, then promise, then price, then automate. Each phase ends at a gate that must pass before the next one starts.

Chargeback only starts once allocation has matched the invoice for a full quarter of showback. The durations are an illustrative reference plan, not forecasts.

Required resources

• A platform observability team of about 4–6 engineers (SRE, data and FinOps), with a finance partner part-time (an illustrative estimate, not a benchmark).

• An OTel and Prometheus-compatible backend sized for GPU-node and machine telemetry, plus lakehouse tables for FOCUS and allocation data.

• NVIDIA Mission Control or an equivalent stack for factory hardware; OpenCost for Kubernetes allocation.

• Edge collector images for Jetson Thor/Orin, validated against control-loop timing (Canvas 7).

Risk assessment

Success metrics

• 100% of GPUs, services and AI-enabled machines emitting labelled telemetry.

• Training goodput reported for every priority job, and rising quarter on quarter.

• Fleet-average SM active above 0.5 on training nodes.

• At least 95% of GPU spend allocated to a team and product, reconciling to invoices within 2%.

• Every tier-1 service has an agreed SLO with burn-rate alerts; error-budget policy used at least once.

• Cost per validated model and AI cost per autonomous machine-hour published monthly.

Open questions and follow-up prompts

Open questions for practitioners

• Which on-machine AI functions are safety-relevant, and therefore get the 99.99% tier and a protected budget?

• What is the agreed definition of a "validated model" for unit-cost reporting (link to Canvas 4 gates)?

• On-prem DGX vs cloud GPU mix: how is on-prem hardware amortized into the cost model?

• Who should own the error-budget decision for a cloud inference API used by both customers and machines?

• What telemetry can customers request under the EU Data Act, and in what format?

• Is a vendor AIOps layer (such as Minerva or Mission Control) worth adopting, or is open tooling alone sufficient?

Follow-up prompts

1. Elaborate on an SLO design for VLA policies in the field, including behavior-quality SLIs beyond latency.

2. Provide a cost-allocation model for a shared DGX cluster running training, Isaac Sim and inference, with FOCUS column mapping.

3. Analyze the challenges of observability for autonomous haul trucks with multi-hour network outages.

4. Suggest an experiment to measure joules per action for GR00T-class models on Jetson Thor across precision modes.

Sources

• Last9 — NVIDIA DCGM Exporter: setup and GPU metrics guide

• NVIDIA NGC — DCGM Exporter container

• NVIDIA — Automating AI factory operations with Mission Control

• NVIDIA — Accelerate token production in AI factories

• Unite.AI — NVIDIA launches Minerva AIOps platform

• NVIDIA NGC — AI Factory Operations Agent

• Equinix — From the cluster to the circuit

• Dash0 — OpenTelemetry GenAI semantic conventions explained

• OpenTelemetry — semantic-conventions-genai repository

• Wongpanich et al. — ML Fleet Efficiency with ML Productivity Goodput (MLSys 2026)

• Google Cloud — Monitor goodput with the ML Goodput Measurement library

• FinOps Foundation — Introducing FOCUS 1.3

• FOCUS specification

• GMI Cloud — FinOps for AI infrastructure

• UD — AI FinOps cost discipline (Aug 2026)

• Cast AI — Kubernetes cost dashboard (2026 report figures)

• OpenCost — 1.121.0 Kubernetes inference cost tracking

• DEV — Kubernetes cost allocation with OpenCost

• Chung et al. — The ML.ENERGY Benchmark

• Chung et al. — Where Do the Joules Go?

• IntuitionLabs — Energy use per AI inference task

• USENIX SREcon26 Americas — Agility Robotics fleet observability slides

• Canonical — Robotics observability at ROSCon 2025

• Freshfields — From reporting to rating: EU data centres

• Beveridge & Diamond — EU data center sustainability ratings

• Global Mining Review — Fortescue's autonomous haul fleet surpasses 2 billion t milestone

• Regulation (EU) 2026/1744 and AI Act timelines: see Canvas 5 sources.