How language models became robot brains

In July 2023, a researcher at Google set up a table with a Coke can and pictures of celebrities. They asked Google's newest robot system, RT2, to move the Coke can to Taylor Swift. The robot slowly picked up the can and awkwardly placed it on the edge of the picture.

It doesn't sound impressive. The movement was clumsy and the task was trivial. But Karol Hausman, one of the researchers on the team, would later describe this as the moment it became clear to him that this was going to work.

Within a year, Hausman and key members of the RT2 team had left Google to form Physical Intelligence. By late 2024, their robots could open padlocks, fold laundry, peel oranges, make grilled cheese sandwiches, and clean up kitchens they'd never seen before. By early 2026, they'd raised over a billion dollars at a $5.6 billion valuation, with reports of a new round at $11 billion or higher.

That clumsy demo answered the field's fundamental question with a shaky but definitive yes: large language models can be trained to be robot brains. Since then, the resulting Vision-Language-Action (VLA) architecture has converged across every major lab and started commoditizing, exactly as LLMs did. We believe the constraint that now decides who wins in Physical AI is not the model. It is data: diverse, action-grounded trajectories from the physical world, and the infrastructure to collect, annotate, and curate them. Almost nobody has built that infrastructure.

Text was the wrong interface

In 2022, Google researchers began exploring what role large language models might play in robotics. Their first systems, SayCan and Inner Monologue, used LLMs as planners, breaking complex tasks into subtasks like "find a sponge, pick up the sponge, go to the spill."

But the LLM couldn't actually control the robot. Once it decided to "pick up the sponge," a completely separate neural network, trained to imitate humans performing small tasks, computed the actual motor signals. The LLM was choosing from a menu of pre-trained behaviors, and the menu was the ceiling: to place a Coke can on Taylor Swift, someone would have had to train that exact behavior first.

The interface between the two systems was natural language, predetermined text instructions the control layer knew how to interpret. That worked for simple tasks. It could never cover the infinite variety of situations a general-purpose robot encounters.

RT2 deleted the interface

RT2 asked a simple question: what if the language model output motor commands directly?

The Google team took their existing multimodal language models, which could already process text and images, and trained them to output robot control signals, using the same human demonstration data that had trained their control networks. The only change was that the language model itself now learned to produce actions.

It worked. More importantly, it generalized. RT2 could solve tasks involving objects and concepts it had never seen in robot training data. The Taylor Swift demo worked not because anyone had trained the robot on Taylor Swift, but because the underlying language model had learned about Taylor Swift from internet-scale pretraining, and RT2 figured out how to connect that abstract knowledge to physical action.

The team coined a term for the architecture: Vision-Language-Action. Three modalities, one model.

π0: a brain and a spinal cord

By early 2024, key members of the RT2 team had reassembled at Physical Intelligence. In October, they released π0, and the improvement over RT2 was remarkable.

π0 takes Pali-Gemma, a 3-billion-parameter open-weight multimodal LLM from Google, as its foundation, and adds a second component: an action expert. That sounds like a return to the old modular approach. The crucial difference is the interface. In the old systems, the LLM told the control network what to do in text. In π0, the action expert shares the same transformer architecture as the underlying Gemma LLM, so the two components effectively think together while retaining the benefits of modularity.

The result mirrors the human nervous system: a "brain" for high-level reasoning and a "spinal cord" for fast motor control. The LLM is the brain, slow and deliberate, processing language and visual information. The action expert is the spinal cord, fast and reflexive, generating precise motor commands.

Technically, the Gemma LLM processes incoming images and text prompts. Each image is broken into 256 patches, each patch encoded into an embedding vector, and the vectors pass through 18 transformer layers whose attention heads learn to connect relevant information. One head, for example, might specialize in finding objects that match words in the prompt: when you ask the robot to "uncap the pen," it learns to connect the word "pen" to the specific image patches containing the pen.

The action expert turns that representation into motor commands with flow matching, the same technique used in AI image generation. Generating a smooth robot trajectory is like generating a natural image: there are many valid ways to uncap a pen, just as there are many valid images of a cat. Flow matching starts from completely random trajectories and iteratively refines them into smooth action sequences at 50 Hz.

The result is a 3.3-billion-parameter model, smaller than the 5–55 billion parameter RT2 models, that runs on a consumer-grade NVIDIA RTX 4090 at 73 milliseconds per inference and can fold laundry, clean kitchens, and manipulate objects it has never seen.

The iteration since has been rapid. π0.5 introduced co-training on heterogeneous data sources (robot demonstrations, web data, semantic subtask prediction, and bounding-box annotations), enabling a mobile manipulator to clean entirely new kitchens in 10–15 minute long-horizon tasks. π0.6 upgraded the backbone to Gemma 3 with an 860M-parameter action expert processing up to 4 camera views at 448×448 resolution. π*0.6 added reinforcement learning from autonomous practice, the robot learning from its own mistakes: 13 hours of continuous espresso-making, and 59 boxes assembled in a real chocolate factory.

In February 2025, Physical Intelligence open-sourced the original π0 weights. The architecture is no longer proprietary.

The architecture is converging

Every major lab has landed on the brain-and-spinal-cord pattern. Google's Gemini Robotics pairs Gemini 2.0 with a fine-tunable action layer. NVIDIA's GR00T N1 makes the analogy explicit with "System 2" (VLM for reasoning at 10 Hz) and "System 1" (diffusion transformer for motor output at 120 Hz).

The open-source ecosystem has closed the gap faster than anyone expected. OpenVLA (7B parameters) outperformed the closed RT-2-X (55B) by 16.5 percentage points in absolute success rate. HuggingFace's SmolVLA showed that a 450M-parameter model can match VLAs ten times its size. VLA submissions at ICLR grew from 1 in 2024 to 9 in 2025 to 164 in 2026.

This is exactly what happened with large language models: the architectures converged, the weights became accessible, and the competitive advantage shifted to data and deployment infrastructure. Which brings us to the problem.

The data wall

VLA models are trained on human demonstrations. Someone teleoperates a robot to uncap a pen, and the model learns to imitate that behavior. The quality, diversity, and scale of this demonstration data directly determine what the model can do.

RT2's breakthrough insight was that internet-scale pretraining gives the model conceptual knowledge: it knows what Taylor Swift looks like, what pens are, how objects generally behave. But that knowledge only becomes useful when grounded in real trajectories from real robots in real environments. And that data doesn't exist at anywhere near the required scale. The combined open-source robotics datasets total roughly 5,000 hours of interaction data. Robotics data cannot be scraped. It must be collected one interaction at a time in the physical world, annotated with action-grounded labels, and curated for diversity.

A landmark paper at ICLR 2025 established the scaling law that matters: policy generalization scales as a power law with the number of training environments and objects, not with demonstrations per environment. Past roughly 50 demonstrations per environment, additional ones have minimal effect. Robotics doesn't need billions of demonstrations. It needs hundreds of thousands of diverse, high-quality trajectories.

Diversity is harder to systematically produce than volume. You need deliberate variation in scenes, objects, backgrounds, and failure modes, and quality control that catches when your data is too homogeneous. A subtle finding: expert diversity can actually be counterproductive. Different human teleoperators introduce velocity multimodality that confounds policy learning, and a distribution debiasing method yielded 15% gains, equivalent to 2.5x more pre-training data.

The architecture problem is solved. The data problem is now the binding constraint on the entire industry.

Five tools, no pipeline

As Foxglove co-founder Adrian Macneil put it: "If you go to a Tesla or a Waymo, there are literally hundreds of people building this kind of tooling inside those companies." Most robotics startups don't have hundreds of people, period.

Yet the typical robotics data pipeline requires stitching together five or six incompatible tools: recording with a custom ROS node or MCAP, storage on S3, visualization in Foxglove or Rerun, annotation in Encord or Scale AI, versioning with DVC, and training pipeline integration with custom code. Each integration point requires custom engineering. Format conversions are error-prone. Companies lose temporal synchronization between camera streams and motor commands during export, then spend weeks debugging why their model performs worse on "identical" data.

VLA annotation breaks existing tools

Traditional computer vision annotation labels static frames with bounding boxes or segmentation masks. VLA training requires action-grounded, temporally coherent, multi-modal annotation. Every visual annotation must be paired with corresponding motor commands, and labels must stay semantically consistent across entire action sequences. If you're annotating "grasp the cup," that annotation needs to span the approach, contact, and lift phases with synchronized motor data throughout.

Tesla pioneered this with 4D labeling, reconstructing scenes from eight cameras across multiple frames simultaneously using neural radiance fields. Musk confirmed they "rewrote all labeling software for 4D. Very different from labeling single photos," calling each 4D labeling effort worth 100x previous approaches because the offline reconstruction can look at future frames, resolving occlusions and generating perfect velocity and acceleration labels automatically.

The tools built for 2D and 3D annotation don't support this. The gap is structural.

No vendor covers the whole pipeline

Scale AI has 100,000+ production hours and a recent partnership with Universal Robots, but focuses on collection and annotation rather than the full pipeline. They don't natively handle MCAP or ROS bags, so you convert your data into their formats and lose the tight sensor synchronization VLA training requires.

Encord positions itself as the top data labeling platform for Physical AI, and does support LiDAR, point clouds, and sensor fusion. But it lacks native robotics data ingestion, on-robot recording, and action-trajectory annotation purpose-built for VLA, and its ontology is designed for autonomous vehicles, not manipulation.

Foxglove created MCAP and raised $58M, and leads in robotics data visualization and fleet management. But it has zero annotation capabilities: you can visualize your data beautifully, but you can't label it. Rerun offers excellent open-source 3D visualization but is code-first, with no platform-level annotation or curation. Labelbox has no native LiDAR annotation support. V7 is limited to visual data, with no 3D or robotics modalities.

The pattern is clear: annotation tools don't understand robotics data formats, robotics tools don't do annotation, and no single vendor bridges recording, visualization, annotation, curation, versioning, and training pipeline integration. Companies either stitch together five tools with custom code or build everything themselves. Both options burn engineering time that should go toward the product.

China is building data factories at national scale

While Western startups each build custom pipelines in isolation, China is treating robot data as national infrastructure. China accounted for 80–90% of global humanoid robot installations in 2025, with approximately 16,000 units shipped. Unitree led with 5,500 units starting at approximately $16,000 each, an order of magnitude cheaper than Western competitors. AgiBot shipped over 5,000 units and built the world's largest humanoid manipulation dataset: over 1 million trajectories across 217 tasks.

More significantly, 40+ state-funded robot data collection centers have been announced across China, with roughly 24 already operational. Beijing's Shijingshan training camp covers 10,000+ square meters with 16 specific scenarios, where workers use VR headsets and exoskeletons to generate movement data at scale. UBTech sold $80M of humanoids to government data centers alone.

This is the EV playbook applied to robotics: standardized infrastructure, large-scale data collection, and sharing across the entire domestic industry. The data gap this creates will compound over time.

The flywheel nobody has built yet

The endgame is clear: deployed robots collect real-world data while working, that data retrains the model, and the next deployment performs better. Stanford's Scanbot demonstrated the loop with a mobile manipulator deployed in a library for two weeks: VLM book identification improved from 32% to 72% with only 26 human interventions over 10 days.

But no robotics startup has the infrastructure to run this loop at scale. Building the flywheel (edge-to-cloud sync, automated quality filtering, distribution analysis, active learning for sample selection, dataset versioning, training pipeline integration) is exactly where companies are stuck. Most teams are still manually copying ROS bags from robots to laptops.

Even the skeptics need the same data

Not everyone is convinced VLA alone is the answer. Yann LeCun argues VLAs "can't truly reason or plan because they lack a model of the world." His JEPA architecture takes a different approach, and Meta's V-JEPA 2, fine-tuned on 62 hours of robot data, achieved 65–80% pick-and-place success on novel objects without any environment-specific training.

The practical consensus is convergence. NVIDIA's GR00T N1.6 already integrates a world foundation model as its VLM backbone, and DeepMind pursues both Gemini Robotics (VLA) and world models through Genie. But the debate doesn't change the constraint: whether you're training a VLA on demonstration trajectories or a world model on observational data, both require massive amounts of high-quality data about how the physical world works. The bottleneck is the same.

What the industry actually needs

Based on what VLA training requires, and what the current vendor landscape fails to provide, the gap comes down to five capabilities no single vendor delivers today:

1. Native robotics data ingestion: MCAP, ROS bags, Protobuf with temporal synchronization preserved through every processing step. Not format conversion into generic image and video pipelines.

2. Action-grounded annotation: Trajectory labeling that pairs language instructions with synchronized multi-camera video and motor commands. Not frame-by-frame bounding boxes designed for autonomous vehicles.

3. Embedding-based curation: Distribution analysis across environments and objects, quality validation that catches synchronization drift and distribution gaps before they become model failures.

4. Fleet-scale data management: Infrastructure for the deployment-to-training loop, from deployed robot recording through automated quality filtering to training pipeline integration.

5. End-to-end traceability: Dataset lineage from raw sensor recording through annotation, review, export, and model prediction. When a model fails in deployment, you need to trace back to which training examples drove that behavior.

The window is now

The Taylor Swift Coke can demo happened in July 2023. Less than three years later, robots fold laundry, clean kitchens, and assemble boxes in chocolate factories. Autonomous vehicles taught the industry that impressive demos don't mean deployment is around the corner: in 1995, a Carnegie Mellon team drove across the US 98.2% autonomously, and it took three decades to get where we are today.

But the model layer is commoditizing on schedule. Physical Intelligence open-sourced π0. NVIDIA released GR00T N1 under Apache 2.0. Google is providing Gemini Robotics access to trusted testers. What's not commoditizing is the data infrastructure to train, evaluate, and continuously improve these models on your specific robot, in your specific domain, at production quality.

That is where the advantage will be won. The companies that get their data flywheel spinning first will compound their lead with every deployment cycle. Everyone else will still be copying ROS bags onto laptops.