Consider what happens when a skilled annotator labels a complex driving scene.
They're looking at a cluster of LiDAR points near the road edge. The shape suggests a person, but the lower body is hidden behind a parked car. They check the camera feed: yes, a visible head and shoulders, with posture suggesting forward motion. They scrub adjacent frames to confirm temporal consistency. They weigh alternatives. A cyclist? No helmet visible, no handlebar signature in the point cloud. Someone standing still? The frame-to-frame motion says otherwise.
After this synthesis, perhaps a few seconds of work for an experienced annotator, they draw a bounding box and select "pedestrian" from a dropdown menu.
The dropdown selection is what persists. The reasoning chain that produced it is gone: the cues that mattered, the alternatives considered, the confidence level, the similar cases that informed the judgment. All of it evaporates at "submit."
We believe that discarded reasoning is training data, and for physical AI, the most valuable kind. It is what end-to-end driving models need to learn nuanced behavior, and it is what regulators will demand when they ask why a model made a call. Every pipeline optimized for labels per dollar is systematically deleting it.
Labels are the residue of reasoning
An annotator classifying a partially occluded object is not pattern-matching. They're synthesizing information across sensor modalities, reasoning about physics and geometry, applying learned heuristics about how the world works, and making probabilistic judgments under uncertainty. That is exactly the cognitive work we want AI systems to learn. Yet our pipelines store only its residue: bounding box coordinates and a class assignment. The perceptual strategies, the considered alternatives, the confidence calibration, the edge case reasoning: all of it is treated as ephemeral context rather than data.
The loss compounds. When a trained model later misclassifies a similar edge case, there's no way to diagnose whether the problem was insufficient training examples, ambiguous guidelines, or flawed reasoning in the original labels. When guidelines change, perhaps after an incident reveals a gap, previously labeled data can't be systematically re-evaluated, because no one recorded how annotators interpreted the old guidelines. When two annotators disagree on a difficult case, we capture the disagreement but not the reasoning divergence that caused it.
Annotation is only the sharpest instance of a general pattern. A support ticket shows "escalated to Tier 3" but not the synthesis of conversation history and customer context behind the escalation. An engineering change shows the code diff but not the architectural reasoning that shaped it. In each case the outcome survives and the intelligence that produced it dies.
Context graphs treat reasoning as data
The emerging fix goes by several names: decision traces, reasoning graphs, context graphs. We'll use "context graphs" here, but the idea matters more than the label: treat the reasoning behind decisions as data worth storing, not context to be discarded.
A context graph differs from a traditional knowledge graph in what it prioritizes. Knowledge graphs model entities and relationships: this person works at this company, this vehicle passed this location at this time. Context graphs add a decision layer on top: who decided what, based on what information, referencing what precedents, with what confidence and caveats.
For annotation, that means structured reasoning captured alongside each label. Rather than just "pedestrian at coordinates (x, y)," the trace might read: classified as pedestrian with 85% confidence based on visible head and shoulders despite lower-body occlusion; considered cyclist but rejected it for absence of helmet or handlebar signature; similar to case #4521, where partial occlusion was resolved through temporal consistency; flagged as an edge case requiring expert review.
That transforms training data from input-output pairs into a record of how skilled humans interpret the physical world: which visual cues matter, which alternatives to consider, how to handle occlusion and uncertainty. Foundation Capital's Jaya Gupta and Ashu Garg argue that the next trillion-dollar platforms won't emerge from adding AI to existing systems of record; they'll be built by capturing the decision traces enterprises have never systematically stored. Applied to physical AI, the implication is direct: the companies that capture annotator reasoning now will own training data that teaches models not just what to perceive, but why.
Systems of record were built to forget
Reasoning has been lost for architectural reasons, not careless ones. Data warehouses like Snowflake and Databricks receive data via ETL after decisions are made; by the time data lands, the context is gone. CRM and ERP systems store current state: what an opportunity looks like now, not what it looked like when the decision was made. Ticketing systems capture outcomes, not the cross-system synthesis that happened in someone's head. Reasoning was never treated as data, so nothing was built to hold it. An annotator's judgment that a LiDAR cluster is a pedestrian rather than a cyclist was considered part of the process, not part of the output.
Four capabilities now make reasoning storable. Event sourcing preserves complete decision histories rather than current state. Graph databases model the relationships between decisions, precedents, and outcomes. Vector embeddings make reasoning traces semantically searchable: "find decisions similar to this one" becomes a queryable operation. And LLM-based agents can capture structured reasoning as a natural byproduct of work.
That last point is the design principle companies implementing context graphs converge on: reasoning capture must emerge as exhaust from valuable work, not as a separate documentation burden. Annotators shouldn't write explanations. They should make decisions through interfaces that record the reasoning as structured data.
End-to-end models and regulators both demand the why
Two forces are converging on the same requirement for autonomous vehicle development.
The first is architectural. The industry is shifting toward end-to-end neural networks that learn driving behavior from data rather than encoding it in explicit rules. Tesla's FSD V12 reduced hundreds of thousands of lines of rule-based C++ code to roughly 2,000, with behavior emerging from networks trained on driving examples. Rivian's newly announced Large Driving Model takes a similar approach, using Group-Relative Policy Optimization to learn driving strategies from fleet data, and it explicitly needs training data that teaches "what not to do" with reasoning context. When a model learns end-to-end from sensor inputs to control outputs, the difference between a pedestrian "preparing to cross" and one "waiting at curb," and the reasoning behind that distinction, is exactly the decision signal it needs. Skilled annotators produce that signal today. Pipelines throw it away.
The second is regulatory. The EU AI Act classifies autonomous vehicles as high-risk systems requiring transparency mechanisms, human oversight provisions, and fundamental rights impact assessments. Automotive suppliers including Bosch and Volkswagen's Cariad software unit expect to require, by 2026, that every AI decision be explainable and traceable. When a regulator asks why the system classified an object as safe to pass, the answer must trace back through model behavior to training data to annotation reasoning. That chain is only as strong as its weakest link, and today the annotation reasoning link is missing entirely.
Agreement metrics measure the wrong thing
Roughly 80% of AI project failures reportedly trace to data quality issues. Yet quality assurance in annotation focuses almost exclusively on agreement metrics: did annotators label the same things the same way?
Agreement is the wrong proxy for the failures that matter. Two annotators might both label something "pedestrian" while holding completely different mental models of what distinguished it from a cyclist or a shadow. They agree on the output and diverge on the reasoning. When the trained model later encounters a case that falls between their mental models, it may fail in ways neither annotator's labels would predict.
Reasoning traces surface this hidden divergence. If annotators make their reasoning explicit, even through structured interface choices rather than free-form writing, systematic differences in interpretation become visible. Guidelines get refined to close ambiguities. Training targets specific reasoning gaps. The labels themselves become more reliable, because the reasoning behind them is queryable and auditable.
There is evidence that captured reasoning pays off at scale. LinkedIn's 2024 implementation of knowledge graphs for customer support parsed tickets into tree structures that preserve reasoning relationships, then connected patterns across issues, improving retrieval accuracy by 77% and cutting resolution times by 28%. The system learned not just that issues were resolved, but how problems get solved, and that institutional intelligence transferred to new situations. An annotation operation with the same memory survives annotator turnover, handles edge cases consistently, and supports far sharper quality assurance.
Edge cases are where deleted reasoning costs the most
The long-tail distribution of driving scenarios makes rare events disproportionately important: a model that handles 99% of situations correctly but fails catastrophically on the remaining 1% is not safe to deploy. Edge cases, the ambiguous and never-before-seen situations, are where human judgment works hardest and where discarding it costs most.
Watch how edge cases are handled today. An annotator encounters a person on an electric unicycle, partially occluded, moving erratically. The guidelines don't address it. They make a judgment call, perhaps "cyclist" as the closest category, and move on. Another annotator classifies a similar case as "pedestrian," since there's no traditional bicycle. A third flags it for review. The disagreement is noted; the reasoning behind each classification is not. When guidelines are eventually updated for personal electric vehicles, there's no systematic way to identify which previous annotations the gap affected, how they were handled, or whether the annotators' reasoning matches the new policy. The institutional learning that should flow from edge case resolution leaks away.
With reasoning captured, each edge case compounds instead. Similar cases cluster by reasoning pattern, not just label outcome. Guidelines evolve from how skilled annotators actually reason through ambiguity. New annotators see relevant precedent automatically: here's how senior annotators handled this situation, and why. Better edge case handling produces more consistent labels, which improve model performance on difficult cases, which surface new edge cases, which enrich the reasoning corpus. The annotation operation becomes smarter over time rather than merely larger.
Sensor fusion is the same gap in concentrated form. When a camera detects an obstacle that LiDAR misses, or radar returns are ambiguous, current annotations capture the fused output but lose the per-sensor assessments and the logic that resolved the conflict. When the system errs, debugging means reconstructing what each sensor saw and guessing why fusion went wrong. If the reasoning were captured (LiDAR downweighted because rain degraded its returns, camera trusted because the object sat within its high-confidence range), diagnosis becomes tractable. The value compounds with sensor complexity: Rivian's new R2 platform combines 11 cameras, LiDAR, 5 radar units, and 12 ultrasonic sensors, which means more potential conflicts to resolve and more reasoning worth keeping.
Capture reasoning as exhaust, not homework
The obvious objection: won't this slow annotation to a crawl? Only if it's designed as documentation. The principle that makes it practical is that reasoning capture should scale with decision difficulty, not apply uniformly to every label.
For routine cases where confidence is high and guidelines are clear, capture almost nothing: a confidence indicator, a scenario classification. The system learns to identify routine cases from annotator behavior and stays out of the way for the bulk of the work. For edge cases (ambiguous scenarios, guideline gaps, annotator disagreement), the annotator is already thinking carefully; capturing that thinking adds modest effort and generates disproportionate value.
Structured interfaces replace free-form writing. Rather than "explain your reasoning," the system prompts: which visual cues were most relevant? What alternatives did you consider? How confident are you? Does this relate to a previous case? The answers are structured, queryable, and comparable across annotators. And cross-references to similar past decisions turn the graph into searchable precedent, so consistent edge case handling no longer depends on every annotator independently rediscovering policy. The context graph becomes institutional memory that outlives individual contributors.
Volume or judgment
Companies building physical AI face a fork in how they treat training data. The conventional path optimizes volume and throughput: maximum labeled examples at minimum cost per label. That path scaled the labeling industry, and it systematically discards the one thing that makes human judgment worth paying for. The other path treats labels with reasoning traces as a different product from labels alone, because in safety-critical applications, where edge cases determine deployment readiness, the reasoning behind annotations decides whether the labels can be trusted, defended, and audited at all.
Regulation is removing the choice. Once explainability requirements bind, "the model classified it this way because of the training data" stops being an acceptable answer, and training data that can't explain itself becomes a liability. The question for data infrastructure is no longer "how do we label more data faster?" It is "how do we capture the reasoning that makes labels meaningful?" We think context graphs (decision traces stored as structured data alongside annotation outputs) are the answer, and that in physical AI, where mistakes have real consequences, the teams that keep the why will beat the teams that keep only the what.
At Avala, we're building data infrastructure that captures not just labels but the reasoning behind them. If you're thinking about how annotation quality shapes deployment safety, we'd like to hear what you're seeing.
