There's a moment that happens on almost every robotics project. The team has collected a substantial dataset, trained a promising policy, and watched it work in the lab. Then it fails in ways that don't make sense: it handles scenarios it should find difficult while stumbling on cases that seem trivial. Dig into the data and the causes surface: timestamps that drifted, calibrations that shifted, labels that contradicted each other across batches.
Every one of those problems was avoidable. Not with better tools or more sophisticated algorithms, but with sustained engineering attention at the points where data enters the system. The team didn't lack capability; they lacked bandwidth. Their engineers were focused on perception and planning, the core research problems, while data quality issues accumulated in the background.
Here is the claim this post defends: that attention cannot be bought by the ticket. Data infrastructure thrown over the wall to a vendor fails no matter how good the vendor is, because the failures live in context a ticket cannot carry. The fix is embedded engineering: engineers who sit in your Slack, push code to your repo, attend your standups, and understand not just what data is needed but why it's needed and how it fits into the larger technical goals.
Tickets buy labor; robotics data needs context
The transactional model works for well-defined, stable tasks. If you need a thousand images labeled to clear guidelines, a ticket queue is fine: crisp boundaries, unambiguous success criteria, no deep system context required.
Robotics data pipelines are not like this.
Consider sensor calibration. On paper it's straightforward: measure the geometric relationships between sensors so data can be transformed into a common coordinate frame. In practice it's a moving target. Sensors drift as temperature changes. Mounts loosen over time. Someone bumps the LiDAR during maintenance. The calibration that was accurate last week might be introducing centimeters of error today.
The ticket version: the team notices LiDAR and camera data aren't aligning, files a ticket, and the vendor recalibrates and closes it. A week later the problem reappears because no one set up continuous monitoring. Another ticket gets filed.
The embedded version: the engineer responsible for calibration notices alignment drift in routine data reviews before it becomes a visible problem, because they've instrumented the pipeline to flag degradation automatically. When a significant drift occurs, they don't just fix it: they investigate why it happened and adjust the monitoring to catch similar issues earlier. Over time, calibration problems become rare because the system has been hardened against them.
The difference isn't capability; it's context. The embedded engineer knows how calibration errors propagate through downstream processing, which failure modes the perception team cares about most, and how to prioritize by actual impact rather than ticket severity scores. They're not solving abstract problems; they're solving yours.
Context compounds; tickets reset to zero
Watch what happens when an engineer stays embedded with a team over months. In the first week, they're learning the codebase and operating from first principles. After a month, they know why design decisions were made, which shortcuts have caused problems before, and which team members to consult for which questions. After six months, they're a repository of institutional knowledge: they remember the calibration incident from February, they know which failure modes correlate with weather changes or shift schedules, and their intuition about where to look first is usually right.
None of this transfers through documentation. It lives in the engineer's head, built through direct experience with a specific system. With transactional vendors, the context is lost at the end of every engagement. The next vendor starts from scratch and makes the same mistakes the previous one made, because there's no continuity.
This is the most underappreciated argument for the embedded model. A great vendor might solve any individual problem faster than a great embedded engineer. But over a year, the embedded engineer has solved a hundred problems, each solution informed by the previous ninety-nine. The vendor solves each of those hundred problems independently, and the hundredth one takes just as long as the first.
Here's what that compounding attention looks like in practice: five lessons about where robotics data pipelines break.
Lesson 1: Instrument the ingestion layer obsessively
Pipelines start with ingestion: getting raw sensor data from robots and capture rigs into a form that can be processed. It sounds mechanical, but it's where many subtle problems originate.
The lesson, learned sometimes painfully: tag every data segment with sensor health indicators, calibration state, environmental conditions, and collection context. When a policy trained on that data behaves unexpectedly, you can query for all training examples collected while a particular sensor was degraded, calibration was outside acceptable bounds, or operators reported environmental anomalies. Debugging turns from guesswork into investigation: instead of wondering whether data quality might be the problem, you check directly whether the bad behaviors correlate with specific data quality issues. Often they do, and the fix becomes obvious once you can see it.
Teams that skip this step land in a painful place six months later: a model that misbehaves, a large dataset they don't fully trust, and no efficient way to find the contaminated parts. Retroactively adding instrumentation is possible but expensive. Doing it from the start is cheap insurance.
Lesson 2: Synchronization errors compound silently
Robots carry sensors that sample at different rates and latencies. A camera might capture frames at 30 Hz with 10 milliseconds of shutter delay. A LiDAR might sweep at 10 Hz with points acquired over 100 milliseconds. An IMU might sample at 200 Hz with negligible latency. Aligning these streams precisely is surprisingly difficult, and the errors stay invisible until you try to train a policy on the data.
That's the insidious part: the data looks fine on casual inspection. A 50-millisecond offset between camera and LiDAR is invisible when you review individual frames. But when a model tries to learn the relationship between visual appearance and depth measurements, those 50 milliseconds mean the objects in the image don't quite match the objects in the point cloud. The model learns a fuzzy, uncertain relationship between vision and depth, not because reality is fuzzy, but because the data is slightly misaligned.
Continuous synchronization monitoring that detects drift in real time is essential, and flagging degradation isn't enough: diagnose the cause. A hardware issue? A software buffering problem? A network delay that appeared when the system topology changed? Root cause is what prevents recurrence rather than patching symptoms.
Lesson 3: Technical validation isn't enough; review data like a researcher
Raw data that passes technical checks can still be useless for training. Maybe the teleoperation was sloppy. Maybe the scene was unrepresentative. Maybe the task was completed in a way that doesn't generalize.
Regular quality reviews surface these issues before they contaminate training sets, and they're surprisingly high-leverage. Worth looking at: intervention rates (how often did operators have to take over from autonomy, and why?), drift metrics (did the robot's behavior change over a collection session in ways that suggest fatigue or distraction?), and coverage (is the data representative of the deployment distribution, or clustered around easy cases?).
These reviews produce weekly reports that make progress measurable: data quality improving over time, systemic issues that need process changes, problems caught early enough to fix before they affect training. Without them, teams discover data quality issues only when a model performs poorly, the most expensive possible time to learn. It's not glamorous work, but it determines whether your training data teaches what you think it teaches.
Lesson 4: The evaluation harness is the highest-leverage build
The most valuable work happens at the intersection of data collection and model evaluation. When training and evaluation share the same data infrastructure, you can replay exact scenarios from your training set and compare policy behavior to ground truth, identify the specific examples a policy fails on and examine them in detail, and build regression test suites from historical failures so new policy versions don't regress on previously solved cases.
The key is wiring evaluation into existing workflows rather than treating it as a separate activity. When someone says "the policy fails on scenario X," they should be able to pull up scenario X exactly as it was recorded, examine it from any angle, and understand precisely what went wrong.
Teams that build this early iterate dramatically faster. Teams that treat evaluation as an afterthought spend most of their debugging time reproducing failures rather than understanding them.
Lesson 5: Close the loop between evaluation and collection
One of the phrases we use internally is "closing the loop," and it's worth spelling out what we mean. In a typical robotics workflow, data collection, labeling, training, and evaluation are separate stages connected by handoffs, and every handoff is an opportunity for inconsistencies to creep in. Closing the loop makes the stages continuous and bidirectional. When evaluation reveals a failure, it flows directly back into collection priorities. When labeling hits an ambiguous case, the resolution updates the guidelines and triggers re-review of past labels. When training produces unexpected results, the investigation traces through to specific data quality issues, calibration states, or collection conditions.
Embedded engineers are the connective tissue that makes this work. Present at each stage (ingestion, labeling, and evaluation), they catch the problems that only become visible when you look across stages. That matters most when things go wrong: in a disconnected pipeline, a problem at any stage can take weeks to diagnose because each team sees only its piece. In a closed loop, the engineer who sees the symptom can trace it back through the entire chain.
The teams that scale treat data infrastructure as engineering
The same pattern recurs from early-stage startups with a handful of robots to companies operating fleets across multiple facilities. The teams that struggle most often have the most sophisticated research and the least investment in data infrastructure. They build impressive policies in simulation and get demos working in controlled conditions. But when they try to scale, data quality issues multiply faster than they can fix them: every new robot, environment, and sensor configuration introduces variation the ad hoc pipeline wasn't designed to handle.
The teams that progress fastest treat data infrastructure as a first-class engineering problem from early on. They instrument before they strictly need to, set quality standards that seem excessive at their current scale, and build systems that will still work with ten times as much data from ten times as many sources. The upfront cost is higher; the payoff is that scaling doesn't break everything.
One caveat these pipelines have earned: they are complex systems that fail in surprising ways, and anyone who claims to have solved all the problems is either working on toy scenarios or not looking closely enough. The best any team can do is catch problems quickly, make them easy to diagnose, and learn from each failure so the same mistake doesn't happen twice.
That's the choice, stated plainly. Buy tickets and you buy the same fix a hundred times. Embed engineers and the hundredth problem gets solved with everything learned from the ninety-nine before it.
If any of this resonates, reach out, not for a sales pitch, but to compare notes on what's working. These problems are hard, and everyone in the trenches has something to teach.
