A robot reaches for a wine glass that's wet from condensation. The gripper makes contact, applies force, and lifts. The glass slips 3mm before the controller compensates. The robot recovers, completes the lift, and places the glass successfully.
How do you label that trajectory?
Was it a success? Technically yes: the glass reached its destination. But a model that only sees "success/failure" labels will never learn to anticipate slip or modulate grip force preemptively. The richness of what happened during that grasp (the contact dynamics, the force profile, the recovery behavior) gets flattened into a binary.
This is the annotation taxonomy problem, and it is the quiet bottleneck in Physical AI. The constraint is not how much manipulation data the field collects. It is that robotics has no shared language for saying what that data means. Every lab is inventing one privately, incompatibly. Until someone builds the standard publicly, nobody's data compounds.
The primitives don't exist
A bounding box either contains an object or it doesn't. Computer vision annotation has had 15 years to mature around primitives like that: ImageNet established image classification conventions in 2009, COCO standardized object detection, segmentation, and keypoint annotation in 2014. The formats are fixed, inter-annotator agreement metrics are standardized, and the tooling is sophisticated.
Robotics annotation has none of this.
The largest open robotics dataset, Google DeepMind's Open X-Embodiment, uses RLDS format (Reinforcement Learning Datasets). RLDS captures observation-action pairs: at timestep t, the robot observed X and took action Y. That is sufficient for basic imitation learning, but it provides no schema for quality. There is no standardized way to indicate that this grasp was stable while that grasp was marginal, no convention for marking contact events, force profiles, or task phase transitions.
The frontier labs aren't filling the gap. Physical Intelligence's RECAP methodology depends on advantage labels, annotations indicating whether a particular action was better or worse than what the baseline policy would have taken, but PI hasn't published their annotation schema. Figure AI mentions "VLM-generated hindsight labels" for their Helix model, labels produced by a vision-language model (VLM) that capture what happened rather than how well it happened. NVIDIA generates synthetic trajectories at scale without addressing how those trajectories get validated for physical plausibility.
Everyone is solving the annotation taxonomy problem internally. Nobody is sharing their solutions. The result: every team reinvents the same primitives, poorly.
Five primitives every team is reinventing
To see why this keeps getting reinvented, consider what a complete annotation schema for manipulation data has to capture: at minimum, five primitive types, none of which has an agreed definition today.
Task phase segmentation. A pick-and-place operation isn't one action. It's a sequence of phases (approach, pre-grasp positioning, grasp execution, lift, transport, place, retract), each with its own objectives and failure modes. A grasp can fail during lift (insufficient force), transport (collision), or place (poor alignment). Labeling only the final outcome discards the diagnostic information about where in the sequence things went wrong.
Grasp taxonomy. Not all grasps are equivalent. A power grasp wraps multiple fingers around an object, maximizing contact area and friction at the cost of precision. A precision pinch uses fingertips only, trading grip strength for dexterity. A lateral pinch suits thin objects like cards or keys; a three-jaw chuck suits cylinders. Grasp type determines required force, stability under perturbation, placement precision, and recovery options, and a model trained without grasp type annotations will struggle to select strategies for novel objects.
Contact state. Contact is not binary. A gripper transitions through no contact, initial contact, partial contact, full contact, slip onset, and slip, and each state calls for a different control strategy: speed during approach, force control at first touch, increased grip or abort at slip onset. Force/torque sensors can detect these transitions, but the raw data still has to be segmented and labeled. A model that can't distinguish contact states moves jerkily or fails to respond to perturbations.
Force profiles. Appropriate force depends on the object: perhaps 2N of grip force for a ripe tomato, 50N for a metal tool. An egg might crack at 10N. A wet glass needs more force than a dry one to prevent slip, but not so much that it shatters. Force profiles also encode skill: an expert demonstrator ramps force smoothly, a novice spikes it abruptly. Labeling profiles allows filtering demonstrations by quality.
Failure modes. How a manipulation fails matters as much as whether it failed. Missing the object entirely is a perception failure. Contact without a stable grasp is a grasp planning failure. Dropping during lift is a force control failure, colliding during transport is a motion planning failure, and placing incorrectly is a precision failure. Each points to a different fix, from camera placement to tactile sensing to finer motor control. A model trained on undifferentiated "failure" labels can't learn any of these distinctions.
A 50-millisecond error corrupts the label
Even a rich schema fails if labels land on the wrong timestep. A typical manipulation trajectory mixes RGB and depth from multiple cameras at 30Hz with proprioceptive state, end-effector pose, and force/torque readings at 100 to 1000Hz, plus gripper state and tactile arrays at 100 to 500Hz. All of these streams must be temporally aligned before annotation can begin.
A 50-millisecond misalignment between camera frames and proprioceptive state can corrupt action labels entirely: if the observation is from t=1.00s but the action label is from t=1.05s, the model learns the wrong association. And alignment is an engineering problem in its own right. Sensors have different latencies, networks introduce jitter, recording systems may not share a clock, and timestamps can be inconsistent across modalities.
Contact events make this acute. An instantaneous event labeled at frame 150 of the camera stream must correspond to the correct samples in a proprioception stream with 10x the temporal resolution. Miss, and the model learns incorrect relationships between visual observations and contact dynamics.
Manipulation quality has no ground truth
The deeper problem: even perfectly aligned, schema-complete labels require judgment calls that experts disagree on. Consider two successful grasps of the same object. Grasp A approaches directly, grasps quickly, lifts smoothly, and places accurately, in 3.2 seconds. Grasp B approaches cautiously, repositions twice before grasping, lifts with a slight wobble, and places with minor adjustment, in 5.8 seconds.
Which is better? A high-throughput warehouse wants Grasp A's efficiency. A home robot handling fragile heirlooms wants Grasp B's caution: the repositioning suggests the demonstrator was handling uncertainty, a strategy that may generalize better to cluttered scenes and variable objects. There is no objectively correct answer, only context-dependent tradeoffs.
That makes quality a schema problem, not just a philosophical one. How do you train annotators to rate quality that is context-dependent? How do you measure inter-annotator agreement when reasonable experts disagree? How do you version schemas as your understanding of quality evolves? Physical Intelligence's RECAP sidesteps the question by computing advantage values relative to the current policy, but that requires running the policy to get baseline predictions and comparing against human demonstrations or corrections. It's computationally expensive and still requires ground truth about what "better" means.
The auto-labeling trap
The obvious response to this complexity is automation: if human labeling is slow and expensive, have VLMs generate labels. Figure AI's Helix model takes this approach; its hindsight labels describe what happened in a trajectory. "The robot picked up the red cup and placed it on the shelf." That is useful for language-conditioned policies that need to associate trajectories with natural language descriptions.
But auto-labeling fails on exactly the categories that matter most. Task phase boundaries require understanding robot intent, not just observable state: the transition from approach to pre-grasp positioning is a shift in objective, invisible to a VLM watching continuous motion. Contact states require force information no camera captures: you cannot see whether a grasp is stable or on the edge of slip. And grasp quality judgments require physical intuition about contact mechanics, friction, and object properties that current VLMs haven't learned.
VLMs still earn their keep on coarse task descriptions, object identification, anomaly flagging, and first-pass annotations for humans to refine. But the fine-grained labels that separate production-grade manipulation data from demo-grade data require human expertise, or at minimum human validation.
What a COCO for manipulation would specify
The robotics community needs what ImageNet and COCO provided for computer vision: standardized schemas that let datasets be combined, benchmarks be compared, and tooling be shared. Nothing about this standard is mysterious. It would define the primitive types above: grasp categories, contact state transitions, task phases with success conditions, force thresholds, and failure diagnostics. It would fix temporal conventions for events that span timesteps, multi-rate sensor data, and uncertainty in event timing. It would carry quality metadata: demonstrator expertise, annotation confidence, inter-annotator agreement, and known ambiguities in the labels. And it would define extension mechanisms so domain-specific categories and new schema versions don't break compatibility.
This is infrastructure work. It won't produce impressive demo videos or benchmark-topping model results. It is also the kind of foundation that separates mature fields from emerging ones.
The cost of waiting compounds
The Vision-Language-Action (VLA) model explosion is creating urgent demand for manipulation training data. Physical Intelligence, Figure AI, Skild, NVIDIA, Google DeepMind, and dozens of smaller players are all training general-purpose manipulation policies, all facing the same annotation challenges, and all solving them independently, incompatibly. The longer this continues, the more technical debt accumulates: datasets with idiosyncratic schemas can't be combined, and models trained on one team's annotations can't be fine-tuned with another team's data.
Autonomous driving already ran this experiment. Early AV datasets used different label formats, coordinate systems, and object taxonomies. Combining them required extensive preprocessing, and transfer learning was complicated by schema mismatches. Standards eventually emerged, but only after years of duplicated effort and wasted engineering time. Physical AI can skip that detour. The annotation taxonomy problem is solvable. It requires investment in infrastructure rather than models, coordination rather than competition, and the recognition that some problems are better solved once, together, than repeatedly, in isolation.
The stakes are concrete. Teams that get manipulation annotation right won't just have better training data. They'll have data that compounds: datasets that can be combined, versioned, and extended as the field matures. Teams that don't will own archives of trajectories nobody else can use.
The robot grasping the wet glass needs more than a success/failure label. It needs a language precise enough to capture the slip, the recovery, and the physics of contact. Whoever builds that language builds the foundation everything else in Physical AI stands on.
