We just got back from the Actuate Robotics conference in San Francisco, and one theme surfaced in nearly every conversation: the bottleneck for embodied intelligence is no longer hardware or model architecture. It's evaluation.
That claim runs against instinct. The hard problems in robotics are supposed to be hands that grip delicate objects, legs that balance on uneven terrain, policies that generalize across tasks. Those are real problems, and progress is being made on all of them. But from time spent with teams at the frontier, we keep relearning the same lesson: the gap between a working demo and a deployed system is rarely about making the robot more capable. It's about proving the robot is capable enough, reliably enough, to be trusted in the real world. Robotics has a proof problem, not a capability problem.
That's evaluation. And right now, the infrastructure for it barely exists.
The Physical Turn: mistakes stop being recoverable
For the past decade, AI has lived in a domain where mistakes are recoverable. If a language model generates a bad paragraph, you regenerate. If an image model produces an artifact, you try again. The cost of failure is measured in compute cycles and user frustration.
We're now entering what we've started calling the Physical Turn: an era where AI inhabits machines that move through the real world. Humanoid robots, warehouse automation, autonomous vehicles, agricultural robots, surgical assistants. These systems don't just process information; they take actions that affect physical reality. A language model that occasionally produces nonsense is annoying; a warehouse robot that occasionally drops heavy objects is dangerous. The margin for error collapses when AI operates alongside humans and valuable infrastructure.
Capital is following the transition: industrial leaders are talking less about "AI features" and more about "AI-powered fleets." But the friction point isn't where most of that money expects it to be.
The demo-to-deployment gap is an evaluation gap
Over the past year, we've been embedded with humanoid startups and industrial automation teams, watching how they develop and validate their systems. The pattern repeats: a compelling demo is easy; a robust, safety-critical deployment is hard. And the gap between the two is almost always evaluation, not imagination.
Take a warehouse picking system. A team trains a policy with imitation and reinforcement learning, and after a few months the robot picks a variety of objects in the lab and handles reasonable variation in lighting and position. The demo video looks great. Investors are excited. Customers are interested. Then comes deployment, and the questions turn uncomfortable. How does the policy handle objects it has never seen? Lighting that differs from the lab? A conveyor belt whose speed varies, objects at the edge of the workspace, a human reaching unexpectedly into the work area? What's the failure rate over ten thousand runs in production, not ten runs in the lab? And when failures occur, is the consequence a dropped item, a damaged product, or a safety incident?
The demo answered none of these questions. It showed that the robot can work; it didn't show that the robot will work, reliably, under the full range of conditions it will encounter.
Software solved this problem long ago: no serious engineering organization ships code without unit, integration, regression, and performance tests. Robotics has no equivalent. Most teams evaluate by watching a few runs, deciding it "looks pretty good," and hoping for the best. That was enough when robots did repetitive tasks in controlled environments. It fails for general-purpose systems in unstructured environments, where the risk surfaces span equipment damage, worker safety, regulatory compliance, and customer trust.
What the frontier already agrees on
Our conversations with engineering leaders across humanoids, mobile manipulation, and industrial automation kept circling back to three convictions.
Simulation is mandatory, not optional
To validate a robot under thousands of conditions, you can deploy fleets into the real world and learn through trial, error, and expensive downtime, or you can build high-fidelity digital worlds where models fail safely and cheaply. Everyone is converging on the second option. Real-world testing is slow, expensive, and risky: rare edge cases might occur only once in ten thousand hours of operation, and some failures are dangerous enough that you don't want them to happen even once on real hardware.
In a well-built simulator, you run thousands of experiments overnight, systematically vary lighting, object positions, sensor noise, and timing to map where a policy works and where it fails, and reproduce failures exactly to verify that fixes hold. The catch is that these digital worlds are genuinely hard to build and maintain. The physics needs to be accurate enough that policies transfer from simulation to reality: the infamous "sim-to-real gap." Sensors need realistic emulation, including their failure modes, and environments need to capture the variation policies will meet in deployment.
But the direction is clear. We're moving from "train in the world, test in sim" to "train and test in sim, then confirm in the world." The digital world becomes the gate every policy must pass before it touches physical reality.
Evaluation must be offline-first
"Vibes-based" evaluation is dead for any serious deployment. A new policy doesn't get loaded onto a robot; it first runs through an offline pipeline of logged data from past deployments, simulated scenarios covering known edge cases, and regression tests derived from previous failures.
That pipeline has to answer specific, quantitative questions. What's the failure rate across long task horizons, not just first attempts? Many policies succeed at a single pick but accumulate errors over hundreds. How does performance shift under variations in lighting, weather, clutter, or object distribution? Does the new policy regress on failure modes you already fixed? If the answers aren't quantitative and repeatable, the policy doesn't move forward. Training gets you a promising model; evaluation decides whether that model ever sees the real world.
Environments, not just datasets
The traditional paradigm (train on a dataset of inputs and labels, evaluate on a held-out test set) works for perception. It's insufficient for robotics, because robots don't just perceive; they act, and their actions change what they perceive next. To evaluate a policy, you can't score its predictions on static data. You have to let it interact with an environment and observe the consequences over time.
Foundation model labs and robotics teams are converging on the same realization: raw video alone won't get you to robust general-purpose behavior. They want structured reinforcement learning environments that capture real tasks, constraints, and rewards; high-fidelity logs for offline RL, imitation learning, and safety analysis; and environment APIs, not static datasets, so policies can interact, explore, and be stress-tested. The market is shifting from data consumers ("give us more video") to environment inhabitants ("let us live inside your world model, with realistic tasks and feedback"). Partners who can supply both environments and grounded data, tied to real robots, real sensors, and real outcomes, are in urgent demand.
Evaluation is only as good as its ground truth
Underneath all of this sits a more fundamental problem. If you don't know what "correct" behavior looks like, you can't measure how close your robot comes to achieving it.
For perception, ground truth is straightforward: label the images, measure how often predictions match. For a system that acts over time, it's harder. There are many valid ways to pick an object and multiple acceptable paths around an obstacle; there is no single right answer to score against.
The practical answer is human demonstration. Record expert teleoperators performing tasks, capturing what they see, what they do, and what outcomes they achieve, and those demonstrations become the standard trained policies are measured against. This is why teleoperation and egocentric capture matter so much: they don't just teach the robot what to do, they define what success means.
This is where we've pointed Avala's work. We deploy field teams with industry-standard capture rigs to collect first-person manipulation data: calibrated multi-camera egocentric video, synchronized robot state and force/torque readings, and task-level annotations with success/failure signals. That solves the cold-start problem for imitation learning: instead of starting from random exploration, labs plug into rich human demonstrations in the exact conditions they care about. We also provide teleoperation-as-a-service, where trained operators pilot robots through complex tasks using VR, haptic, or traditional control setups. Those sessions produce training data for new policies, grading data for evaluating existing ones against human baselines, and structured logs of exactly where the robot needed help.
The field has no common ruler
Every lab reports success rates on its own internal tasks, using its own definitions and metrics. When one team reports a "99% success rate" on its picking task and another reports a "95% success rate" on theirs, you have learned almost nothing. The tasks differ. The objects differ. The definition of "success" might be more or less forgiving. The numbers are nominally about the same thing and actually measure incommensurable quantities.
The cost is field-wide: progress is hard to track, benchmarks aren't comparable, customers can't compare vendors, and researchers can't build on each other's work. What's needed is shared benchmark infrastructure: standardized task definitions, canonical object sets, fixed evaluation protocols, and common metrics.
We're building this into Mission Control. We treat LiDAR, radar, camera, and IMU as views into a single 4D world model (3D geometry and semantics evolving over time), with multi-sensor fusion aligning modalities into a consistent frame. On top sit evaluation layers: task suites for manipulation, navigation, inspection, and human-robot interaction, with canonical hold-out sets, standardized metrics, and fixed difficulty tiers. The goal is that someone can say "our robot improved by 15% on standardized benchmark X" and have that statement mean something outside their own building.
Safety regression: unit tests for robots
The most important application of all this infrastructure is safety regression testing. In software, when a bug is discovered and fixed, you write a test that would have caught it, and every subsequent change runs against that test. Over time, the suite becomes a comprehensive encoding of everything the team has learned about what can go wrong.
Robotics needs the same discipline. When an operator intervenes, an object drops, or a collision nearly occurs, that failure should become a permanent test case, and every future policy version should be verified against every historical failure. The bar for deployment should continuously rise.
We're making this a first-class concept in Mission Control. Every teleop session and autonomous run is mined for interventions: where did an operator step in, what incipient failure did they prevent, what signals were involved? Interventions get distilled into scenario definitions, things like "forklift forks misaligned with pallet under low light" or "humanoid slips on wet floor while carrying a box." Each scenario becomes a reusable test case, replayed in high-fidelity simulation against multiple model checkpoints, with domain randomization around the original failure conditions: what happens under exactly these circumstances, and under slightly varied ones (brighter lighting, a different approach angle, a faster movement speed)?
Every new policy must pass all previously logged failure cases before deployment. Every new real-world incident becomes a new test, raising the bar for the next release. Reality continuously hardens your simulation, and simulation continuously protects your deployments.
Prove it works, or it stays in the lab
The transition to embodied AI won't be built on raw pixels alone. It will be built on trusted, verifiable data pipelines that connect real-world experience, high-fidelity digital worlds, human expertise, and automated, repeatable evaluation. The teams making the fastest progress already operate this way: they invest in evaluation infrastructure before it feels necessary, build regression suites from their first failures, and demand quantitative answers about reliability instead of more impressive demos.
We think the bottleneck is evaluation, and we think the solution is ground truth: captured carefully, organized rigorously, and tested relentlessly. The robots that make it from lab to deployment will be the ones that can prove they work, not just show that they can. If you're working on this problem too, we'd love to hear from you. Getting evaluation right is what will make the Physical Turn actually happen.
