An automotive OEM came to us with a narrowly scoped request: improve labeling quality to reduce flickering detections in their perception stack. We took the job and ended up disproving the premise. Flickering is not a labeling problem. It is an architecture problem, and the frame-by-frame labeling pipeline this OEM ran (along with most of the industry) makes it close to inevitable. No amount of labeler discipline or tracking cleverness fixes an error the pipeline manufactures upstream.
That diagnosis changed everything that followed. A pilot to patch one data quality issue became a multi-year infrastructure partnership built on a single architectural move: fuse every sensor stream into one 4D representation of the world before any label is drawn. The names and some details are anonymized, but the substance is real, and the path from "fix this specific issue" to "build infrastructure that prevents these issues" is one we've walked with several partners now.
Flickering was the symptom, not the disease
The OEM's perception team called the problem "flickering reality." An object (a pedestrian, another vehicle, a piece of debris) appears in one frame, disappears in the next, then reappears two frames later in a slightly different position. The object is really there; the tracking is unstable. The system can't maintain a consistent representation of the world across time.
For a demo, flickering is annoying. For a production ADAS (advanced driver assistance system), it's a safety problem: planning and control inherit every moment where the vehicle's understanding of the world doesn't match reality.
The team had traced the flickering to three interrelated causes. Camera and LiDAR streams drifted out of temporal sync by a few milliseconds, enough that objects appeared in different positions depending on which sensor you asked. Calibration between sensors degraded as mounts shifted and temperatures changed. And the labeling pipeline, which processed camera and LiDAR separately before merging them, amplified those small errors into visible artifacts.
They'd tried the obvious fixes: tighter synchronization requirements, more frequent recalibration, better tracking algorithms. Each helped a little. None held, because frame-by-frame labeling (label each image independently, then stitch the labels together across time) was fighting them. Every slightly different labeler judgment about object boundaries, every sync error that shifted a detection by a few pixels, handed the tracker inconsistent inputs.
Our first weeks were diagnostic. We embedded with their perception team and traced specific incidents backward: a pedestrian that flickered in and out of existence over 20 frames, followed through the calibration state, the timestamp anomalies, and the annotation differences that produced it. The unglamorous detective work confirmed the pattern. Calibration drift, temporal sync, and labeling inconsistency were interconnected in ways that made them impossible to fix independently. Improve calibration, and frame-by-frame labeling still produces discontinuities from labeler disagreement. Improve labeling consistency, and drifting calibration still writes errors into the labels that aren't the labelers' fault. The whole system had to change together.
Label the world, not the frame
The core of our solution was to stop labeling frames and start labeling the world.
The traditional approach treats each sensor frame as an independent unit: label the camera image for frame 100, then the LiDAR point cloud for frame 100, then move to frame 101 and repeat, and hope the downstream stitching decides that the bounding box in frame 100 corresponds to the one in frame 101. It's simple to distribute and parallelize. It's also a false picture of the world. Objects persist through time; a pedestrian in frame 100 is the same pedestrian in frame 101. Labeling frames independently asks the system to rediscover a persistence you could have encoded from the start.
Our alternative fused all the sensor data (camera, LiDAR, radar, and IMU) into a unified 4D representation before any labeling happened. We call this the 4D World-Data Engine. The "4D" is literal: three spatial dimensions plus time, treated as a continuous whole rather than a sequence of disconnected snapshots.
In this representation, objects aren't bounding boxes in individual frames. They're persistent entities in space-time, observed by different sensors at different moments. Label the entity once and the label projects correctly onto every frame where it's visible, from every sensor that observed it. The labeler's job changes from "draw a box around this object in this frame" to classifying an entity that already exists in the world model. And because the entity's position is determined by fusing all sensor evidence, calibration and sync errors get resolved during fusion rather than propagating into labels.
The Neural Digital Twin made debugging deterministic
Implementing this meant building what we call a Neural Digital Twin of the OEM's data: a continuous reconstruction of everything that happened during each drive. Raw sensor data goes in; a coherent, temporally consistent world model comes out, representing the geometry of the world at each moment, the positions and motions of all objects, and the sensor observations that would result from any viewing angle.
Building the twin means estimating the precise calibration state at each moment, resolving temporal offsets between sensors, and fusing dense camera images, sparse LiDAR points, and noisy radar returns into one consistent representation, at the scale of thousands of hours of driving data. The key insight is that problems which are hard frame-by-frame become tractable with full temporal context. Calibration drift is invisible in a single frame; across hundreds of frames it shows up as a systematic bias, objects consistently shifting in one direction between camera and LiDAR.
The capability the OEM's team found most valuable was deterministic replay. Any scenario could be replayed through the twin with exact fidelity: go back to the timestamp of a flickering incident and examine exactly what each sensor saw, how the fusion interpreted it, and where the problem originated. Better still, the replay could be modified to test hypotheses. What if calibration had been correct? What if the timestamps had been perfectly synchronized? The twin simulated these counterfactuals, which made it possible to prove a fix would work before implementing it in the real pipeline. Debugging stopped being guesswork.
Edge cases became regression tests
Before, when the perception team hit a flickering incident, they'd investigate it, fix the specific cause, and move on. Nothing guaranteed the same problem wouldn't recur, or that the fix didn't introduce regressions elsewhere.
With the twin, every edge case became a reusable test case. Each identified incident was captured as a deterministically replayable scenario and added to a growing regression library. Every pipeline change (calibration algorithms, fusion parameters, labeling guidelines) had to pass the full library before deployment. Solved problems stayed solved, and the library became institutional memory: everything the team had learned about failure modes, encoded into automated verification.
On top of that we implemented failure mining: automatically scanning new data for conditions that resembled known failure patterns. Calibration metrics drifting toward a problematic range, or synchronization showing unusual variance, flagged a scenario for human review before it caused visible problems. The team moved from reactive debugging to prevention.
Embedding beats the ticket queue
The technical solution was only half of what made this work. From the beginning we embedded a Forward Deployed Engineer with the OEM's perception team, not as a consultant who showed up for weekly meetings but as a functional team member: in their Slack, in their repos, at their standups.
We hold this as a general position: for problems this entangled, embedding beats outsourcing. A ticket queue would have solved the problems the OEM knew to ask about and missed the ones they didn't know they had. Being present in the day-to-day let us see patterns their team had gone blind to, and knowledge flowed both ways: our engineer learned their domain, and their team absorbed our expertise in data infrastructure.
Two practices built the trust that later made a long-term partnership an easy conversation. Weekly reviews ran on actual telemetry (intervention rates, calibration stability metrics, temporal synchronization accuracy, labeling consistency scores), so improvement showed up in the numbers rather than slide decks, and regressions got caught and investigated together. And every pipeline change was reviewed jointly: no black-box handoffs, every modification to ingestion, calibration, fusion, or labeling documented, discussed, and approved, so the OEM's team could maintain the system after our engagement ended.
The pilot became the data stack
Flickering incidents dropped substantially as calibration drift and sync errors got caught upstream. Labeling consistency improved because labelers worked against a stable world model instead of trying to maintain consistency across independent frames. And the perception team could trace any model behavior back through the training data to specific scenarios in the twin, which made debugging tractable in ways it hadn't been before.
The result that surprised us most was organizational. As the OEM saw the value, they routed more of their data through the 4D World-Data Engine. Other teams, facing related problems with different sensor configurations, asked to use the same infrastructure. The narrowly defined project became a core part of their Physical AI data stack, and the one-time engagement became a long-term infrastructure relationship, with our embedded engineers maintaining the pipeline, onboarding new data sources, and improving the system as the OEM's needs evolved.
One message from a senior engineer on their perception team, after a particularly productive few months, stuck with us:
"I really wanted to thank you, Emal, and the Avala team for doing such a phenomenal job. You are all so dedicated and care so much about making sure every little detail is correct and are so responsive... Many of us on perception, including me, have requested that we continue working with Avala for all our other projects. I hope that happens!"
That matters more than any metric. It means the partnership worked on a human level, not just a technical one.
Design the pilot to expand
None of the expansion was accidental. We structured the initial engagement so that success led naturally to broader adoption: the 4D World-Data Engine was built to accommodate whatever sensors and scenarios came next, not just the pilot's. When other teams asked to use it, the architecture was ready.
This is the position we'll defend: recurring data quality problems are usually infrastructure problems wearing a disguise, and teams that keep fixing symptoms will be fixing them forever. The teams that pull ahead invest in foundations even when it feels like overkill for their current scale. The foundations are what let you scale.
If your team is fighting data quality problems that keep coming back, we'd be interested in talking. Not every situation calls for the system we built here. But tracing problems to root causes, and asking what infrastructure would remove them, is worth doing even when the answer differs from ours.
