Evaluation you can trust is not a stage you bolt onto the end of a robotics data pipeline. It is a property of the representation the whole pipeline runs on. Teams understand the individual pieces (teleoperation, sensor fusion, labeling, evaluation); what they underestimate is the handoffs between them, where data changes form, quality silently degrades, and artifacts become indistinguishable from results.
This walkthrough follows one teleoperation session from the moment an operator picks up the controller through to evaluation, and argues that every stage should operate on a single 4D representation of reality. Watch the handoffs: that is where pipelines built as separate systems lose the ability to tell you whether a measured difference is real.
Capture: the policy learns whatever you record
Everything begins with a human demonstrating a task.
Imagine an operator at a control station, wearing a VR headset and holding controllers that map to a robot arm in a warehouse several miles away. The operator sees what the robot sees: a bin of objects to pick and place into shipping containers. They reach, grasp, lift, move, release. The robot mirrors their motions in real time.
Whether that recording is worth training on is decided right here, by three things.
Kinematic isomorphism. The controller should match the structure of the robot it commands. If the robot has a seven-degree-of-freedom arm, the controller should let the operator drive all seven degrees of freedom directly, respecting the robot's actual joint limits and dynamics. Many setups instead use generic VR controllers that track hand position in Cartesian space and rely on inverse kinematics to compute joint angles. That translation layer causes problems: it can fail near singularities, produce dynamically unstable solutions, and oscillate when the solver flips between answers to small adjustments. All of those artifacts end up in the recorded trajectory, and a policy trained on them learns them. It oscillates near certain positions, pauses unexpectedly, and absorbs the quirks of the control interface rather than the structure of the task. When the controller matches the robot's kinematics, the trajectory is a clean expression of the operator's intent.
Synchronization. A session generates multiple streams that must be captured together: control signals, proprioceptive feedback from the joints (positions, velocities, torques), video from cameras on and around the robot, and other sensors such as force/torque at the gripper or depth. Each stream has its own timing characteristics. If timestamps drift even by tens of milliseconds, the recorded gripper force no longer corresponds to the moment of contact visible in the video, which confuses any system trying to learn the relationship between actions and outcomes. Alignment should happen at capture, on a common time base, not as a post-processing step.
Metadata. Context rides along with the streams: who the operator was, what task they performed, environmental conditions, whether the session succeeded, and notes on unusual events. This is what you reach for when curating datasets or explaining why a policy behaves a certain way.
The output of this stage is an instrumented recording: temporally aligned streams of control, proprioception, and perception, annotated with metadata. This is the raw material everything else builds on.
Fusion: one model that explains every observation
Raw streams are not yet in a form suited to training or evaluation. Each sensor has its own perspective, coordinate frame, and noise characteristics. Fusion turns them into a unified representation: a single model of the world that explains all the observations.
We've written separately about how 4D fusion works in detail. For this walkthrough, the key steps are calibration refinement (correcting drift since the last physical calibration by finding cross-sensor correspondences), temporal alignment (removing residual timing offsets using events that should appear simultaneously in multiple streams), geometric reconstruction (fusing observations into a 3D scene with uncertainty estimates), and temporal integration (connecting reconstructions across time so objects persist through occlusion and viewpoint changes).
The result is a Neural Digital Twin: a continuous, queryable model of everything that happened during the session. You can ask what any sensor would have seen at any moment, from any viewpoint, including viewpoints that were never captured. You can query the position and state of any object at any time.
The handoff is the point: raw, multi-stream sensor data goes in; a single coherent world model comes out. Everything downstream (labeling, training data extraction, evaluation) references this model, not the raw feeds.
Labeling: one label in 4D replaces hundreds of frames
In a conventional pipeline, annotators work frame by frame: bounding boxes, classes, traced boundaries, each frame an independent task. If the same object appears in frames 100 through 200, it might be labeled by different people on different days, each making slightly different judgments about the boundaries.
On the world model, annotators label entities that exist in 4D space-time. Identify the box the robot is picking up, assign a class (cardboard box), properties (dimensions, estimated weight), and task annotations (this is the target object for this pick). That single annotation projects onto every frame where the box is visible, from every camera, at every timestamp.
This is more efficient, but efficiency is the lesser benefit. The projection onto frames is computed, not manually created, so the box cannot be labeled differently in frame 150 than in frame 175. Policies trained on conventionally labeled data learn that boundaries are fuzzy, identity is uncertain, and the world flickers. Policies trained on 4D labels learn cleaner relationships because the labels represent cleaner ground truth.
The world model also makes annotations practical that frames cannot express. Contact events (the gripper touching an object, an object meeting a surface) get a precise timestamp instead of a hunt for the exact frame. Intent annotations (decision points, hesitations, corrections) live naturally in a representation that captures full trajectories rather than isolated snapshots.
Extraction: format becomes a projection, not a commitment
In a conventional pipeline you commit to a training data format upfront (images with bounding boxes, point clouds with segmentation masks, trajectory sequences in a particular schema), and labeling produces exactly that. Train a different kind of model later and you relabel.
When training data lives in a 4D world model, the format question is deferred. Extraction is a projection operation. Want images with 2D bounding boxes? Project the 3D entity annotations onto camera frames. Point clouds with semantic labels? Sample the world model from the LiDAR viewpoints. Action-conditioned video for a vision-language-action model? Extract trajectory segments with their visual observations. Same representation, no relabeling.
The benefit compounds: you can combine sessions collected with different sensor configurations, as long as all are represented in 4D, and every improvement to the world model (better fusion, better tracking, corrected calibration) automatically improves all the training data extracted from it.
Evaluation: replay makes failure reproducible
The conventional approach is to run a trained policy on held-out test data and compute metrics. But two questions are surprisingly hard to answer. Is the test data really comparable to the training data? If it was collected at a different time, with different calibration, through a different pipeline version, performance differences may reflect data artifacts rather than genuine capability. And can you reproduce a failure? If the test data is frames in a database, often not.
Replay on the 4D world model answers both. Given a recorded session in 4D form, run an autonomous policy through the same scenario: it observes what it would have observed (the same objects, configurations, and dynamics) and takes actions, which you compare against the human demonstrator or any other baseline.
Because replay happens in the world model, it is deterministic. Run the same policy through the same scenario twice and you get the same result. When a policy fails, you can examine exactly what it observed at the moment of failure, exactly what action it took, and exactly how that diverged from what a good policy would have done.
Replay also supports counterfactuals within the bounds of what was observed: different lighting, an object positioned slightly differently, probing the edges of policy behavior without collecting new data. And metrics computed on replay are directly comparable to training data statistics, because both are computed on the same representation. Apples to apples.
Closing the loop: failures decide what you collect next
The most important part of the system is what happens after a failure.
In a closed-loop system, failures do not just get logged; they feed back into the process that produced them. A failing scenario becomes a test case: a specific, reproducible situation the current policy cannot handle, added to a regression library that future versions must pass. It also sets collection priorities. If the policy struggles with picks at a certain angle or a specific kind of occlusion, that scenario becomes a target for future teleoperation sessions, and the resulting data flows through fusion and labeling into training. The next policy version is explicitly trained on what the previous one failed.
This is what closing the loop means. Collection is informed by evaluation. Evaluation feeds back into collection. Training targets known weaknesses. And it works only because evaluation happens on the same representation as training: identify a failure and you can trace it back to the training data that produced the behavior, examine what the policy learned, and see what different data might teach it. The connection between training and evaluation is not abstract; it is concrete and queryable.
The payoff: when you change something, you know whether it worked
Step back: teleoperation capture, sensor fusion, labeling, training data extraction, and evaluation, all operating on a single 4D representation of reality.
When these are separate systems connected by handoffs, you are never quite sure whether a difference you observe is real. Did the policy improve, or did the evaluation data get processed differently? When everything runs on one scaffold, those questions have answers: the policy got better because it performs better on the same scenarios, represented the same way, and the failure mode is real because you can replay it and watch it happen again.
That confidence is what engineering iteration runs on, and it is what safety-critical applications demand: specific evaluations on specific scenarios with specific outcomes, traceable and reproducible. A pipeline built from separate systems ends up measuring its own seams. Build every stage on one representation, and you measure the policy.
For deeper dives on individual stages (how 4D fusion works, how labeling works at scale, how failure mining feeds back into collection), we've written about each separately.
