Robotics has a seductive idea: train your policy in a physics engine, run millions of episodes overnight, and deploy the next morning. The idea has attracted enormous investment, and simulators like NVIDIA Isaac Sim, MuJoCo, and PyBullet keep getting better. Policies trained in them can walk, jump, manipulate objects, even play soccer. Then they meet reality. The robot that gracefully navigated a simulated warehouse stumbles over a slightly uneven floor. The gripper that picked objects flawlessly fumbles a transparent bottle. The walking policy that looked smooth in the engine produces jerky, unstable gaits on real hardware.
This is the Sim-to-Real Gap, and we believe it is structural, not a temporary engineering deficit that better renderers will close. Simulation converts compute into an approximation of data, and the conversion is lossy in exactly the places where general-purpose robotics is hard: contact, deformation, light, and true novelty. If that is right, the durable advantage in robotics belongs not to whoever builds the best simulator but to whoever industrializes the collection of real-world data.
Why the math looks unbeatable
To see why simulation is so appealing, look at the economics of robotic learning. Frontier models (Vision-Language-Action (VLA) architectures like Google's RT-X or the networks powering Tesla's Optimus) require hundreds of thousands of successful demonstrations to generalize even modestly beyond their training distribution. A robot trained to pick up a red mug in a well-lit lab will often fail on a blue mug in a dim kitchen unless it has seen thousands of variations of mug-picking under diverse conditions.
Collecting those demonstrations in the real world is slow and expensive. You need physical hardware, physical environments, and physical humans to operate the robots. You can't scrape the internet for manipulation data the way you can for text or images; the data has to be generated one demonstration at a time.
Simulation promises to break that bottleneck. Thousands of episodes in the time of one real demonstration. Endless programmatic variation in lighting, object positions, and textures. Dangerous scenarios without hardware damage. Parallelism limited only by GPU budget. If simulation generates 1000x more training data at 1/100th the cost, why would anyone bother with reality?
Because of what simulation actually provides versus what robots actually need.
Simulators are wrong where robotics is hardest
Simulators model physics with equations that are computationally tractable, which means they simplify. The simplifications are invisible in demos and catastrophic in deployment.
Consider contact physics: what happens when a gripper touches an object. In the real world this involves friction, deformation, and micro-slips that depend on surface textures, material properties, and geometry. When a robot picks up a soft rubber duck, the duck deforms. The friction depends on how hard the gripper squeezes, which affects the deformation, which affects the friction. Standard rigid-body engines approximate these feedback loops poorly. They reduce friction to coefficients, struggle with deformable objects like cables and clothing, can't accurately model granular materials like rice or sand, and handle fluids poorly or not at all.
These errors compound. A policy learns a grip that relies on friction properties the simulator gets slightly wrong, and the grip fails on real hardware. Not because the policy is fundamentally flawed, but because it was optimized for physics that don't quite match reality. Optics fails the same way: simulators struggle with transparency, reflections, and refractions, so a perception system trained on simplified rendering fails on a glass bottle or a metallic surface.
Then there is "true novelty." A simulator can only present scenarios that were programmed into it or procedurally generated from known parameters. It cannot prepare a robot for the unknown unknowns of the real world: a sudden gust of wind, a greasy patch on the floor, a chaotic optical reflection, the unpredictable behavior of a house cat. Real environments contain an effectively infinite variety of edge cases that no simulator can fully anticipate.
Policies learn the simulator, not the world
Even where the physics is reasonably right, a deeper problem remains: machine learning systems are extraordinarily good at finding patterns in training data, including patterns that exist only in the simulator. Any correlation between observations and rewards gets exploited, even when it doesn't reflect real physics. These spurious correlations are invisible until deployment, when the policy suddenly fails because the regularities it relied on don't exist in the real world. Researchers call this "overfitting to the simulator": the policy becomes exquisitely tuned to rendering quirks and physics approximations rather than learning behaviors that transfer.
The field has real countermeasures. Domain randomization varies simulation parameters widely during training. System identification tunes simulator parameters against measured real-world behavior. Sim-to-real transfer methods fine-tune simulated policies on real data. These techniques narrow the gap. They do not eliminate it. For simple tasks in controlled environments, cheap approximations of reality may be good enough. For complex manipulation in unstructured environments, the kind of general-purpose robotics everyone is trying to build, the approximations break down in ways that are difficult to predict and expensive to debug.
Moravec's revenge
The gap has deep roots. In the 1980s, Hans Moravec, Rodney Brooks, and Marvin Minsky articulated something counterintuitive: it is comparatively easy to make computers exhibit adult-level performance on intelligence tests or chess, and difficult or impossible to give them the skills of a one-year-old at perception and mobility. High-level reasoning requires relatively little computation; low-level sensorimotor skill requires enormous resources.
In modern AI, Moravec's Paradox manifests as a data problem. It has proven easier to teach an AI to pass the Bar Exam than to teach a robot to reliably clear a dinner table without breaking a glass. Language is discrete, symbolic, and abundant: the internet supplies effectively unlimited text. Physical interaction is continuous, high-dimensional, and scarce. There is no "Common Crawl" for robotic manipulation.
This asymmetry is what makes simulation so appealing, and so insufficient. We have essentially unlimited compute but limited access to physical reality, and simulation promises to convert one into the other. But the conversion is lossy, and for the hardest problems in robotics, "almost the same" as real interaction isn't good enough.
There is no substitute for ground truth
If simulation alone can't produce robust real-world performance, real-world data must. Policies need to see real physics, real lighting, real variation, real edge cases. They need ground truth: data that captures reality as it is, not as a simulator approximates it to be.
That turns data collection economics into robotics economics. Teams that can collect real-world data efficiently (at scale, with consistency, across diverse conditions) gain a compounding advantage. Their policies see more reality, converge faster, and fail less in deployment. They iterate on reality instead of debugging the gap between simulation and it.
This is why we've invested heavily in infrastructure for real-world data collection. Not because simulation is useless, but because you can't ship general-purpose robots on simulation alone. At some point you have to bridge the gap, and the choice is whether you do it efficiently, with infrastructure built for the purpose, or expensively, through trial and error in deployment.
Teleoperation makes reality scale
The most effective way to collect ground truth for robots is teleoperation: humans remotely controlling robots to demonstrate tasks. Teleoperation data is kinematically valid by definition. If the human could make the robot do it, the robot can physically do it. Unlike passive video of humans using their own hands, which captures trajectories a robot may not be able to execute, teleoperation yields verified state-action pairs that imitation learning can directly consume.
The historical objection was cost. Traditional interfaces (master-slave puppet arms, VR controllers, SpaceMice) are expensive, unintuitive, or both; operators fatigue quickly and collection rates stay low. Recent systems change that. GELLO, a kinematically isomorphic controller that is essentially a 3D-printed miniature of the robot arm, gives operators a 1:1 mapping: the operator moves their elbow, the robot moves its elbow. That cuts cognitive load and produces smoother, more human-like trajectories than abstract controllers. Portable systems like UMI (handheld grippers with cameras that use visual SLAM to track position) collect demonstrations without a robot present at all; an operator walks around a kitchen or warehouse demonstrating tasks, and the trajectories are later retargeted to different robot morphologies.
Together these make real-world data collection industrial. Instead of a few engineers in a lab, teams of trained operators can generate thousands of hours of data per month. The economics shift, and real-world data becomes scalable.
Use simulation; don't bet the roadmap on it
None of this means abandoning simulation. It is excellent for early exploration and algorithm development, where fast, cheap iteration matters. It is useful for pre-training policies before real-world fine-tuning, and for stress-testing against edge cases you can define in advance. What it cannot do is supply the ground truth policies need to generalize robustly.
The teams making the fastest progress treat this as an infrastructure problem, not just a research problem: real-world collection capacity with quality control, pipelines from collection through training, and workflows that combine simulated pre-training with real-world fine-tuning. They have recognized that the gap isn't going away. Simulators will keep improving, but so will the complexity of the tasks we ask robots to perform.
Who wins if this is right
If the gap is structural, four things follow.
Real-world data becomes a strategic asset. Algorithms can be copied or reverse-engineered; datasets built through months or years of real-world collection are genuine moats.
Collection economics compound. A team collecting demonstrations at $30 per hour instead of $150 can afford 5x more data on the same budget, and over years that becomes a fundamentally different robot.
Quality matters as much as quantity. Noisy data, inconsistent demonstrations, and poorly calibrated sensor streams actively hurt: policies learn the noise. Collection infrastructure has to be built for consistency, not just throughput.
And teams that treat data as a first-class concern, with infrastructure for collection, processing, and curation, will move faster than teams that focus only on algorithms.
We're still early in the Physical Turn: the transition of AI from the digital realm into embodied systems that act in the physical world. Simulators will keep getting better, and robots trained only in them will keep stumbling on the first uneven floor. The robots we're trying to build will have to learn from reality, and the companies that build the infrastructure to capture reality efficiently will be the ones that ship them.
