Worker retention is a data quality problem. The structure of your annotation workforce directly affects the consistency and reliability of the data it produces, and when you get it wrong you see it in your models: inconsistent behavior, mysterious regressions, policies that work in some contexts and fail in others for reasons that are hard to diagnose.
The question of who creates training data, and under what conditions, is usually filed under ethics. It belongs there. But it is also a technical question, and the industry's default answer, anonymous workers cycled through platforms with no continuity, guarantees churn on exactly the tasks where churn does the most damage. This post is about what we've learned building an annotation workforce we call "coworkers" rather than contractors, and why that distinction shows up in the AI systems trained on their output.
Churn erases the context your labels depend on
The dominant model for AI data annotation is anonymous by design: companies post tasks to platforms where workers they will never know complete them, and the next batch goes to whoever is there. The workers are interchangeable by design. If one person stops working tomorrow, another fills the gap.
This works for simple tasks. If you need to classify whether images contain cats or dogs, it doesn't matter whether the person labeling image 5,000 is the person who labeled image 1. The criteria are unambiguous; any attentive person produces the same labels.
The tasks that matter most for modern AI systems aren't like this. Consider evaluating whether an AI assistant's response is helpful, harmless, and honest. "Helpful" depends on context: a response that helps an expert confuses a beginner. "Harmless" involves judgment calls about misuse, cultural sensitivity, and downstream effects. "Honest" covers not just factual accuracy but appropriate uncertainty and resistance to manipulation.
An annotator encountering these concepts for the first time produces different judgments than one who has spent months discussing edge cases with colleagues and developing calibrated intuitions. When your workforce churns constantly, you lose that context every time someone leaves. Each new annotator starts from scratch, building intuitions that evaporate when they move on. Your labeled data ends up reflecting many different interpretations of your guidelines, all mixed together, with no way to tell which labels came from annotators operating with full context and which came from newcomers.
Models trained on this data learn the inconsistency. They learn that "helpful" sometimes means one thing and sometimes another, that boundaries are fuzzy in ways that reflect annotator variance rather than intentional policy. The flickering you see in model behavior, responses that handle similar cases differently, often traces back to training data that encoded inconsistent human judgments.
Every departure deletes institutional memory
Organizational theory calls it institutional memory: the accumulated knowledge, practices, and context that lives in the people who work somewhere. When experienced employees leave, they take it with them. If turnover is high, it never accumulates; the organization is perpetually starting over.
Annotation workforces have institutional memory too, though it's rarely discussed in these terms. A team that has labeled robotics data for one company for two years knows the platforms intimately, knows which failure modes are common and which are rare, and recognizes when a scenario touches the company's safety priorities. They've seen thousands of examples and developed intuitions about what "normal" looks like, which makes them better at spotting anomalies. They remember when the guidelines changed six months ago and why, which ambiguities have been resolved, and which are still open. When a new example resembles a past edge case, they apply the established interpretation consistently.
This memory is almost impossible to transfer. Documentation captures explicit knowledge, not the tacit understanding that comes from experience. A new annotator reading the guidelines gets the words; an experienced annotator has the context that makes those words meaningful.
The anonymous model destroys this memory continuously. The people who labeled last month's data may have no overlap with the people labeling this month's. Quality becomes unpredictable: some batches are excellent, others inconsistent, and there's no way to know in advance which is which, because it depends on the invisible variable of who happened to be working that day.
Anonymity guarantees churn by design
The anonymous model is characterized by the absence of a relationship: no training beyond the task, no path to harder work, no connection to the company whose data you're labeling, no reason to come back. That structure is efficient for handling variable demand with minimal commitment, but it guarantees churn. Workers leave as soon as something better appears, which means the workers who stay are often those who haven't found alternatives, not those who are most skilled or engaged.
Flexibility is not the problem. Our coworkers work flexibly, from wherever they are, and are paid for accepted work rather than for hours; that is how most of them want to work. What retention requires is the rest: fair pay, training, paths to more skilled work, continuity on the projects a person knows, and a genuine relationship with the work. People who see a future in the work stick around. People who stick around accumulate expertise. Expertise improves quality. It's not complicated, but it requires investment the anonymous model is designed to avoid.
We call our annotation workforce "coworkers" because that's how we think about the relationship. They're not anonymous task-completers; they're colleagues who focus on annotation rather than engineering or research. They have names, histories, and trajectories within the organization. They're trained on the broader context of why the work matters, and they're encouraged to ask questions, flag ambiguities, and improve the guidelines. This isn't charity. If you want consistent, high-quality annotation, you need people who stick around long enough to become consistent and high-quality themselves.
From clickwork to craft
In the anonymous model, annotation is "clickwork": repetitive, interchangeable, requiring nothing beyond basic literacy and attention. All judgment is assumed to live in guidelines written elsewhere; the annotator is just an executor.
Complex annotation breaks that assumption. Guidelines can describe principles, but applying principles to novel situations requires understanding the intent behind them, and that understanding develops through experience, discussion, and feedback, none of which clickwork supports.
We structure our workforce to develop this understanding deliberately. New coworkers start with simpler tasks where guidelines are relatively clear. As they demonstrate accuracy and insight, they move to more complex work: multi-frame tracking, policy evaluation, safety-critical annotation. Eventually some become validators who review others' work, or technical leads who help debug pipelines and improve processes.
This progression does several jobs at once. It provides a path to more skilled work, which improves retention. It routes the most complex tasks to the most experienced people. It creates a quality ladder where each level reviews the level below. And it turns the work into craft: the best annotators can articulate why they made a particular call, notice patterns that suggest problems with the data or the guidelines, and act as participants in quality rather than passive executors.
Inconsistent annotators train inconsistent models
Does any of this actually reach the model? We believe it does, through several mechanisms.
The most direct is consistency. When the same experienced team labels data over time, using stable interpretations of guidelines, models trained on it learn cleaner patterns. They don't have to learn that a label means X sometimes and Y other times.
This matters especially for reinforcement learning from human feedback (RLHF), which trains models to predict which responses humans prefer. If different annotators with different interpretations contribute to the preference data, the model learns a noisy preference function: it might satisfy annotator A's preferences while violating annotator B's, producing erratic behavior that optimizes for preferences nobody actually holds. A stable, well-calibrated team produces judgments that form a coherent whole. The model learns something closer to a genuine policy than a blurred compromise between conflicting views.
The stakes are user-facing. When you deploy an AI system, you're asking users to trust that it will behave as expected, not just on average but consistently, even in unusual situations. Users who encounter erratic behavior lose confidence, and regaining trust is harder than maintaining it.
The stakes are highest in safety-critical applications. If you're training a model to avoid certain behaviors, the training data must consistently exemplify what to avoid and what to do instead. An annotator hired last week, working without the context for why a boundary exists, will label edge cases differently than a veteran who understands the reasoning. Those differences become inconsistent safety behavior in deployment.
Experienced annotators also handle the edges better. They're less likely to be confused by novel situations and more likely to apply principles appropriately where the guidelines are silent, which means the training data covers exactly the cases that cause models the most trouble. And retention closes the feedback loop: annotators who stick around can see how the models trained on their data behave and recognize patterns that trace back to annotation choices. In the anonymous model, the people who labeled the data never see what the model does with it.
Quality is a function of dignity
There's a phrase we use internally: quality is a function of dignity. Treat people as interchangeable task-completers and they internalize that framing. They do the minimum required to get paid, and they don't invest effort in catching subtle errors or thinking carefully about edge cases, because there's no reward for doing so. Treat people as valued contributors whose judgment matters and they rise to the expectation: they take pride in the work, notice things they weren't asked to notice, and bring their full attention. This isn't universal, but it's a strong tendency.
There's also a selection effect. Fair pay, a path to more skilled work, and a respectful environment attract people with skills and options who wouldn't otherwise consider annotation work. The anonymous model tends to attract workers who have no better options or who are passing through. There's nothing wrong with those workers as individuals, but the pool is different, and the pool sets what's possible.
Cheap labels are expensive data
The obvious objection is cost. Fair pay, training, continuity, and paths to more skilled work cost more than posting tasks to an anonymous platform and paying a few cents per label.
Per label, yes. Per usable dataset, the calculation looks different. Count the engineering time spent cleaning inconsistent data, the retraining when models behave erratically, the debugging when edge cases go wrong, and the risk of systematic biases that only emerge after deployment. We've found that the cost of quality annotation is often dominated by the cost of problems rather than the cost of labor. A small percentage increase in annotation spending that reduces data quality problems by a larger percentage is a good trade, and the improvements compound: better training data leads to better models, which require less post-hoc correction.
Manufacturing learned this long ago: the cheapest supplier isn't the best value when they deliver inconsistent quality or disappear without warning. Smart manufacturers invest in supplier relationships and long-term commitments that align incentives. The people who create your training data are your supply chain, and their reliability is your reliability. Treat them as interchangeable commodity providers and you get commodity results: variable quality, no improvement over time, no ability to handle complexity beyond what's explicitly specified.
The horizon matters too. If you're collecting some data, training a model, and moving on, minimizing cost per label makes sense. If you're building AI systems that need continuous improvement, safety monitoring, and adaptation to new domains, you need a sustainable annotation capability. That capability is your workforce.
We call our approach impact sourcing because the impact extends beyond the AI systems. Well-paid annotation work brings economic opportunity to communities that might otherwise be excluded from the AI economy, and training builds skills that coworkers carry forward. But the business case doesn't depend on altruism. Impact sourcing produces better data, more consistently, at lower total cost once you account for the hidden expenses of the alternative. The ethics and the economics point in the same direction.
What this looks like in practice
New coworkers train before they touch production tasks. They learn not just the mechanics of annotation but the context: what AI systems are, how training data affects them, why consistency matters. As they gain experience, some become validators, some become technical leads interfacing with engineers, and some specialize in domains such as robotics, language models, and safety evaluation.
We track quality metrics at the individual level, not to punish but to identify where support is needed. If someone's accuracy drops, we investigate: are the guidelines unclear? Is the task poorly specified? Quality problems are information, not failures.
And we close the loop between annotation and outcomes. Coworkers see how the models trained on their data behave, which annotations were most valuable, and which edge cases caused problems. That visibility connects daily work to tangible results.
Ground truth requires ground-level care
The term "ground truth" comes from surveying: information collected at ground level to verify what's observed from above. The etymology is apt. Ground truth for AI is produced by people working at ground level, reviewing examples and making judgments, and the quality of that truth depends on how those people are treated, supported, and retained. You can't get reliable ground truth from a workforce treated as expendable.
Teams focus on architectures, scaling laws, and training techniques, and those things matter. But underneath it all is data, and underneath the data are people. If you want ground truth that reflects your policies consistently, start with people who stay long enough to understand those policies deeply. If you want annotations that handle edge cases intelligently, invest in annotators who've seen enough examples to develop real expertise.
Treating people well produces better work, better work produces better data, and better data produces better AI. That chain requires a commitment the anonymous model was designed to avoid, which is why worker retention belongs on your model quality dashboard, not just your HR one.
If you're thinking about how to structure your own annotation operations, we're happy to share more about what we've learned. The details matter, and getting them right pays off in ways that aren't obvious until you see the results.
