In October 2024, Waymo published EMMA, an End-to-End Multimodal Model for Autonomous Driving built on Google's Gemini. A few months earlier, Tesla rolled out FSD v12, replacing 300,000 lines of hand-written C++ control code with a single neural network trained on video from its fleet. These are usually read as model milestones. They are data milestones: end-to-end learning does not shrink the annotation problem, it redirects it toward harder targets, and data infrastructure built for frame-level labels is now a depreciating asset.
To see why, start with what end-to-end replaces. For 15 years, autonomous driving decomposed into modules: perception detected objects, prediction forecast their trajectories, planning generated safe paths, and control executed them. The design was interpretable and testable, but errors compound through the pipeline. A pedestrian detector optimizing for mean average precision doesn't know that missing one particular person will cause a collision three seconds later, and the prediction module can't recover information perception discarded. Each module optimizes intermediate metrics that only loosely correlate with what actually matters: safe, comfortable driving.
End-to-end architectures optimize directly for the outcome. A single network maps sensor inputs to vehicle controls, jointly learning what to perceive, how to predict, and when to act. The gains are not subtle. UniAD, which won CVPR 2023's Best Paper, improved every subtask simultaneously against its modular baseline: tracking up 20%, motion forecasting error down 38%, collision rates down 28%.
What EMMA proves, and what it can't yet do
Waymo's EMMA treats autonomous driving as a visual question answering problem. Raw camera images, navigation commands, and ego vehicle history enter a unified language space; the model, built on Gemini 1.0 Nano, outputs trajectories, 3D bounding boxes, and road graph elements as natural language text, trained end-to-end for driving performance.
Two results matter for anyone building data pipelines. First, chain-of-thought reasoning: rather than producing trajectories directly from perception, EMMA reasons through four hierarchical layers (scene description, critical object identification, driving meta-decisions, then trajectory prediction). That structure improves planning quality 6.7%, and it produces rationales a human can audit: why the model yielded, when it decided to change lanes, which objects it weighed most heavily. Second, positive task transfer: co-trained on planning, 3D detection, and road graph estimation, EMMA improves on all three by up to 5.5% over single-task specialists, while setting the best reported results on nuScenes motion planning and staying competitive on the Waymo Open Motion Dataset.
EMMA is still research, not production. It processes only four frames, limiting the temporal reasoning complex maneuvers require. Camera-only perception, without LiDAR or radar, struggles with depth estimation at range. Inference takes 4.5 seconds per frame against a sub-100ms real-time requirement, and Waymo's production vehicles still run modular stacks with multi-sensor fusion. Closing that gap is less a question of architecture than of what these models can learn from data, which is where infrastructure comes in.
Automation doesn't end annotation, it redirects it
Traditional annotation targeted intermediate representations: 2D and 3D bounding boxes around every vehicle, pedestrian, and cyclist, traced lane markings, classified signs, pixel-level drivable surfaces, and, for LiDAR-equipped vehicles, point clouds with millions of points per frame. Industry estimates put comprehensive 4D annotation at up to 800 person-hours per hour of driving.
End-to-end models can, in theory, bypass all of it, and automation is already eating the old work. Tesla's auto-labeling AI now generates the majority of its training annotations, letting the company reduce manual annotation staff significantly in 2024; Waymo's Auto4D iteratively refines 4D object trajectories from sequential point clouds with 25% less human effort. The tempting conclusion is that labeling is a dying business. It isn't. Four supervision needs replace the old ones, and each is harder than drawing boxes.
Trajectory annotations become primary. End-to-end models learn by imitating expert drivers, so they need precisely labeled driving paths: position, velocity, heading, and acceleration over time. A bounding box tells the model where an object is; a trajectory annotation tells it what good driving looks like in response to that object. The task shifts from "what exists in this scene" to "how should we drive through this scene."
Multi-task coherence becomes critical. EMMA's task transfer works only because supervision is jointly consistent across planning, detection, and road graph tasks. If trajectory labels and detection labels disagree about where objects are, the model receives conflicting gradients; if road graph annotations don't align with actual driving paths, it learns confused representations. Annotations must be holistically consistent, not just individually accurate.
Temporal consistency requirements intensify. Modular perception could tolerate frame-by-frame annotation, each detection standing alone. End-to-end systems learn from sequences, so an object that flickers in and out of annotations, or whose identity changes between frames, injects noise directly into trajectory prediction. The tolerances are tight: a 10-millisecond timing error between camera frames introduces 28 centimeters of position error at highway speeds.
Chain-of-thought supervision is a new category. Interpretable models like EMMA need ground truth rationales alongside trajectories: "yielding for pedestrian at crosswalk," "changing lanes to pass slow vehicle," "stopping for red light despite no cross traffic." As regulatory pressure for explainable AI grows, this supervision moves from differentiator to requirement.
End-to-end makes the long tail harder, not easier
Scale does not solve rarity. The scenarios that matter most for safety (near-collisions, unusual obstacles, edge-case weather) are precisely the ones that occur least. Tesla collects 50 billion miles annually from a consumer fleet of over 6 million vehicles; Waymo accumulates 2 million weekly across 2,500 robotaxis and has logged over 20 billion miles in simulation; Mobileye processes 100 petabytes monthly from customer vehicles. Yet analysis of the CODA corner case dataset found only 1,057 valid edge cases in 1 million data points, and researchers estimate that proving autonomous vehicle safety to statistical confidence requires anywhere from 100 million to hundreds of billions of miles, orders of magnitude beyond current collection. The mathematics of rare events are unforgiving.
End-to-end amplifies the problem. Modular systems could be probed component by component for specific failure modes; end-to-end systems must be evaluated whole, so the entire rare scenario must exist in training data: not just the unusual object, but the complete context that determines correct behavior.
That makes active scenario mining core infrastructure rather than a nice-to-have. Tesla's Data Engine detects when shadow-mode AI decisions diverge from human drivers and triggers targeted collection. Mobileye's natural language search traverses 200 petabytes to find scenarios like "tractor covered in snow" or "traffic light in low sun." Simulation fills part of the remaining gap: NVIDIA DRIVE Sim generates photorealistic sensor streams with perfect ground truth annotations, and Waabi's simulation-first approach uses generative AI to build digital twins of real scenarios, exposing models to millions of edge cases safely. But models trained purely on synthetic data degrade significantly in real-world deployment.
So the effective strategy is orchestration: real-world collection for authentic sensor characteristics and natural scenario distributions, simulation for coverage and adversarial cases, and curation pipelines (uncertainty sampling, diversity metrics, explicit scenario search) that decide which data is worth training on at all. Raw miles are table stakes. Findable, curated miles are the advantage.
Evaluation infrastructure hasn't caught up
The field has been measuring progress badly. Open-loop evaluation on nuScenes, where models predict against recorded data without influencing future observations, has dominated benchmarking for years. But 73.9% of nuScenes consists of "driving straight" scenarios, so models can post near-best results using only ego vehicle status, with no perception at all. In closed-loop evaluation, where model actions affect subsequent observations, leaderboard-topping end-to-end planners exhibit critical flaws in safety-critical scenarios. The models that win benchmarks may not drive safely.
The fixes exist. nuPlan provides 1,282 hours of driving across 73 scenario types in a proper closed-loop framework; NAVSIM's Predictive Driver Model Score metrics filter for challenging scenes rather than averaging across trivial ones. For data providers, this extends the pipeline beyond training: safety-critical scenario databases, curated test suites targeting known failure modes, and adversarial example generation become as valuable as raw training data.
Frame-level annotation is a depreciating asset
The transition is accelerating. Tesla has deployed end-to-end at consumer scale, Waymo has demonstrated it at research level on major benchmarks, and Chinese manufacturers including XPeng and Huawei are pursuing similar approaches. The architectural advantages (unified optimization, positive task transfer, foundation model integration) are too large to ignore.
What changes is where data value lives. Intermediate perception labels become less central; trajectory annotation, sequence-level coherence, and closed-loop evaluation become essential. We expect the providers who build trajectory-centric annotation, multi-task schema coherence, and intelligent scenario curation to enable the next generation of autonomous systems, and the providers who keep shipping frame-level bounding boxes without temporal consistency to watch their value erode.
The end-to-end shift isn't just about better algorithms. It's about fundamentally different data, and the infrastructure that produces it.
Our 4D Vector Space approach was designed for exactly this moment: temporally consistent annotation across modalities, multi-task data pipelines, and scenario-centric curation. If you're building end-to-end systems for autonomous driving or robotics, we'd be glad to show you what purpose-built data infrastructure looks like.
