There's a pattern we see repeatedly when we talk to robotics teams. They've built impressive demonstrations. Their policies work in simulation, and they've gotten promising results on real hardware in the lab. But when they try to scale, to more environments, more edge cases, more robots, everything starts to break down.
The culprit is usually data collection, and the failure is structural. Teams treat collection as a side project, something you do when you need more training examples, not a core engineering system that deserves the same rigor as the perception stack or the motion planner. But data collection is infrastructure whether you treat it that way or not. The only question is whether you're building good infrastructure or bad.
Bad infrastructure is the DIY setup most teams still run. Engineers wire up a few capture rigs in the lab, collect hours of footage on USB drives, and manually push logs into ad hoc labeling queues. Someone writes a script to convert the data into training format. Someone else notices the timestamps are slightly off. A third person realizes the camera calibration drifted two weeks ago and nobody caught it. By the time the data reaches the model, it has accumulated enough small errors that training becomes unpredictable. That can work for a demo. It collapses the moment you need fleet-level generalization or anything approaching safety certification.
Good infrastructure is what the rest of this piece describes: teleoperation, egocentric capture, and 4D world modeling, and the design decisions that determine whether the pipeline scales or collapses under its own weight.
Simulation can't capture what you don't know to model
Simulation has gotten remarkably good. Modern physics engines model contact dynamics, soft bodies, and sensor noise with impressive fidelity, and domain randomization helps policies transfer to reality. So why collect real-world data at all? Because simulation can only vary the parameters you thought to include.
Consider a robot learning to pick up objects in a warehouse. In simulation, you randomize object shapes, weights, friction coefficients, and lighting, and the policy learns to handle variation across those parameters. Deployed, it meets variation you never thought to model. The floor has a subtle texture that affects wheel traction. Workers leave coffee cups in unexpected places. Afternoon sun creates glare that matches no synthetic lighting model. A box that looked rigid deforms slightly under gripper pressure. These aren't exotic edge cases; they're the normal texture of reality that simulation abstracts away. The only reliable way to learn them is to encounter them, in the environments where the robot will actually operate.
Human behavior is the sharpest version of this problem. Simulated humans follow programmed patterns; real humans cut corners, reach across robots to grab things, and get distracted by their phones. Training a robot to be safe around people requires observing actual people in actual environments, not animated characters following motion-capture libraries.
So simulation and real-world data are complements, not substitutes. Simulation explores the space of what you've already modeled; real-world data tells you what you forgot to model. Both are necessary, and both require infrastructure to produce at scale.
Naive teleoperation teaches the interface, not the task
At the heart of any serious collection effort is teleoperation. People have been remotely controlling machines for decades, but the requirements for training data are different from the requirements for, say, defusing a bomb or performing telesurgery. You need trajectories a learning algorithm can actually learn from, and that is harder than it sounds.
Consider a naive setup: an operator moves a robot arm with a game controller or VR rig, and hand motions map to end-effector commands through inverse kinematics. The robot follows along, more or less. But "more or less" poisons the data. Inverse kinematics fails near singularities, where small changes in target position require huge changes in joint angles. Latency between input and response causes oscillation and overshoot. The recorded trajectory carries artifacts that have nothing to do with the task and everything to do with the control interface. Train a policy on this data and it learns the artifacts: it oscillates near certain positions, it pauses where the operator waited for visual feedback, it absorbs the quirks of your control mapping instead of the underlying task structure.
The solution is what we call kinematic isomorphism: design the teleoperation interface so the operator's control inputs match the robot's actual joint space. Instead of commanding Cartesian end-effector positions and hoping IK figures it out, the operator directly controls the joints through a rig that mirrors the robot's kinematic structure. This removes the translation layer that introduces artifacts, so recorded trajectories are physically valid and dynamically smooth. Building these rigs is nontrivial. Each robot morphology needs a matched controller with force feedback so operators can feel contacts and resistance, and the whole system needs calibration and maintenance. The payoff is data that reflects how the robot can and should move, not how a human tried to puppeteer it through an indirect interface.
The human side matters just as much, which is why we operate teleoperation as a managed service in dedicated facilities rather than distributing it to crowdsourced workers. Skilled teleoperation is genuinely difficult: it takes practice to build spatial intuition for a body with different proportions and dynamics than your own, and focus to avoid drifting into sloppy habits over long sessions. Our operators build deep familiarity with specific robot platforms over months and years, are supervised and given feedback, and understand why data quality matters because they're part of an ongoing relationship, not anonymous gig workers completing isolated tasks.
Every session is fully instrumented: the robot's motions, the control inputs that produced them, proprioceptive feedback from joints and sensors, and synchronized video from multiple viewpoints. When a trained policy does something strange, you can trace it back through the training data to specific sessions and see whether the behavior came from the operator or from how the data was processed.
Interventions are the most informative moments you record
Imagine an autonomous policy starting to fail: it's about to drop an object or collide with an obstacle, and a human operator takes over to correct the trajectory. Most pipelines log the intervention as a failure case for debugging and otherwise discard it. The autonomous segment was bad data; the human segment was cleanup.
We think this is backwards. The intervention itself is the most informative moment. The autonomous segment shows what the policy was about to do wrong; the operator's correction shows what should have happened instead. Together they form a natural before/after pair that teaches the boundary between acceptable and unacceptable behavior, exactly the contrastive signal learning algorithms use to refine decision boundaries.
So we treat interventions as first-class data and label them explicitly. When an operator takes over, the system logs why: a safety intervention (the robot was about to do something dangerous), a goal intervention (the robot was pursuing the wrong objective), or a quality intervention (the robot was succeeding, but sloppily). Each type carries different training implications. Over time this builds a dataset of precisely the situations where autonomous policies struggle and human judgment is needed, so you can target training at your weakest areas instead of collecting more examples of what the robot already does well.
Robots in human spaces need human-centric data
Teleoperation gives you robot-centric data: what the robot sees, does, and feels. Robots that operate in human spaces also need human-centric data: what people see, do, and expect. Egocentric capture records human activity from a first-person perspective (head-mounted cameras, wrist-mounted sensors, and inertial measurement units (IMUs) for body motion) so policies can learn from how humans naturally perform tasks.
A robot assisting in a kitchen can learn to pick up utensils and manipulate containers through teleoperation. Egocentric capture shows how humans actually use the kitchen: where they look when searching for ingredients, how they sequence multi-step tasks, what paths they take between workstations, how they handle interruptions. That context is what makes a robot a helpful collaborator rather than an obstacle.
The hard part is collecting this at scale with training-grade quality. Consumer cameras and commodity IMUs are cheap but drift, timestamping across devices is error-prone, and nobody can manually check synchronization across thousands of hours of video. We deploy capture rigs with calibrated multi-camera arrays, synchronized recording, and onboard compute for preprocessing, so data comes back with reliable timestamps, known geometric relationships between sensors, and quality metrics that flag problems. Downstream annotation and training can treat it as trustworthy instead of spending cycles compensating for collection-time errors.
The richest signal comes from pairing egocentric capture with task-level annotation. Knowing that a hand moved from position A to position B isn't enough; you want to know it was reaching for the salt shaker because the recipe called for salt, in service of preparing dinner. That hierarchy (low-level motion, mid-level actions, high-level goals) is what lets policies generalize across contexts instead of memorizing trajectories.
Fuse the sensors before you annotate
All of this data (teleoperation logs, egocentric video, sensor streams from robots and rigs) has to become one coherent representation before training. This is where most pipelines fall apart.
The standard approach processes each stream independently, then merges the outputs. Camera images go through object detection, LiDAR through point cloud segmentation, trajectories get smoothed and resampled, and then someone decides that "this bounding box from the camera probably corresponds to that point cluster from the LiDAR." This late fusion compounds errors through the dataset. Objects flicker in and out of existence as detectors disagree across frames. Tracking IDs swap. The same physical object sits at slightly different positions depending on which sensor you trust. A model trained on this data learns the inconsistencies as if they were facts about the world.
We fuse raw sensor data into a unified 4D representation (three spatial dimensions plus time) before any annotation happens. We call it the Neural Digital Twin: a continuous model of where every object was at every moment, how it moved and deformed over time, and what every sensor would have observed from every viewpoint. Annotating in this representation means labeling the underlying reality rather than one sensor's view of it. A label placed once projects correctly onto every camera angle, every frame, and every sensor modality.
Flickering disappears because objects exist as continuous entities in the 4D world; their presence in any frame is just a projection. Multi-sensor disagreement disappears because the twin fuses all the evidence into a single consistent model instead of reconciling conflicts downstream. The twin also enables what frame-by-frame processing can't: generating novel views from camera positions you never captured, replaying scenarios with variations (faster pedestrian motion, different lighting, moved obstacles) while keeping the physics grounded in real observations, and mining failure cases by running autonomy through the twin to find exactly where policy predictions diverge from ground truth.
Ad hoc collection is cheap the way technical debt is cheap
Teleoperation facilities, capture rigs, 4D world modeling, and quality assurance pipelines require significant investment. Whether that makes economic sense depends on how you think about the lifecycle of data.
In the ad hoc model, each project runs its own one-off capture campaign, processes the data, and moves on; the next project starts from scratch because nothing was designed for reuse. In the infrastructure model, you invest upfront in systems that serve many projects over many years: permanent teleoperation facilities with managed equipment and trained operators, standardized rigs so data from different locations is compatible, pipelines robust enough to run unattended, and a 4D world engine that ingests whatever sensors you have and produces one consistent output format.
The per-project cost of ad hoc collection looks low on paper, but you pay it again and again, and quality degrades every time institutional knowledge walks out the door with a contractor. Infrastructure costs more upfront, then amortizes over thousands of hours of data, quality improves as operators gain experience, and each new project inherits everything the previous projects built.
The hidden cost is quality. Ad hoc collection accumulates subtle problems: miscalibrated sensors, drifting timestamps, inconsistent labeling guidelines. These stay invisible until training behaves unexpectedly, and then diagnosis means forensic work through logs and scripts no one fully understands. Well-designed infrastructure catches problems at ingestion and flags them immediately. Data you can trust is what frees a team to work on research questions instead of archaeology.
Evaluation falls out of the same system
Perhaps the most valuable consequence of the infrastructure approach is that evaluation stops being a separate problem.
In a typical workflow, training and evaluation data come from different sources and pass through different pipelines, with possibly different calibration and labeling standards. When evaluation metrics don't match deployment performance, you can't tell whether the model degraded or the evaluation setup differs from reality in some subtle way. When both share the same infrastructure, the uncertainty disappears. The same operators, equipment, and fusion algorithms produce both, so a metric computed on evaluation data genuinely predicts performance, because evaluation data is training-like data.
Shared infrastructure also enables replay evaluation: run a learned policy through a recorded scenario via the 4D twin and measure exactly where it diverges from what a human operator actually did. That's far more actionable than aggregate metrics: not "success rate is 78%" but "the policy consistently fails at this specific step of this specific task type."
Accumulate a library of hard scenarios (intervention events, near-misses, edge cases) and you have regression tests. Every new policy version runs through the full library before deployment, and a regression on a previously solved case is a flag to investigate before shipping. This is the embodied AI equivalent of unit testing in software, and it's only possible when the data infrastructure is coherent enough to support exact replay and comparison.
What to ask a data partner
If you're considering an external partner for robotics data collection, the revealing questions are operational, not commercial. Who owns and operates the facilities? Infrastructure rented per project behaves differently from infrastructure that's permanently staffed and maintained. How are operators trained, and what's the retention rate? Operators who stay for years accumulate expertise that's impossible to replace. How is kinematic validity enforced during teleoperation? If the answer involves IK and hope, the trajectories will have artifacts. Can they deliver world-aligned 4D assets, or just raw video and CSV files? Unfused, unvalidated data costs you engineering time downstream.
The deeper question is whether the partner understands that data collection is a systems problem. Cheap capture is easy; useful capture is hard. The difference is infrastructure.
The advantage compounds
The teams making the fastest progress in embodied AI share one trait: they treat data as seriously as they treat models. They invest in data infrastructure early, when it still feels like overkill. They hold quality standards that look expensive until you price training on corrupted data. And their systems compound: each hour of new data is worth more because it integrates with everything that came before.
This work doesn't produce papers or demos. But the robot that works in the wild, not just in the lab, is the robot trained on data that actually represents the wild. Build the collection system well and it serves you for years. Build it poorly and every project inherits the last project's problems. Either way you're building infrastructure. The only choice is which kind.
If you're thinking through these tradeoffs for your own team, we're happy to share what we've learned. Reach out, and let's compare notes.
