How do you supervise a system that's smarter than you? The question isn't hypothetical. GPT-4 scores in the 90th percentile on the bar exam, and coding assistants produce solutions that human reviewers sometimes can't fully evaluate without running them. These are early systems: the capability gap is widening.
This is the scalable oversight problem, and the field mostly treats it as an open research question. We think that framing is out of date. The leading techniques, debate, recursive reward modeling, process-based supervision, and weak-to-strong generalization, developed at OpenAI, Anthropic, DeepMind, and in academia, are reasonably well specified on paper. What none of them has is a data supply. Each demands oversight data that is structurally different from RLHF preference labels, and most teams' annotation pipelines cannot produce it. Scalable oversight is an infrastructure problem wearing a research problem's clothes.
The stakes are concrete. Teams whose pipelines produce only preference labels will be able to read the oversight literature but not run it. The teams that can generate debate transcripts with verified outcomes, step-level process annotations, decomposed evaluations, and calibrated confidence judgments will be the ones that keep their systems aligned as capabilities scale.
Why "humans rate outputs" stops working
Standard RLHF (Reinforcement Learning from Human Feedback) works like this: humans compare pairs of model outputs and indicate which they prefer, and the model learns a reward function from those preferences. It's the core technique behind the helpfulness and harmlessness of current chat models, and it carries an implicit assumption: humans can reliably evaluate which output is better. As capabilities increase, that assumption breaks in three ways.
Complexity. Some outputs are too complex to evaluate in reasonable time. A thousand-line codebase, a detailed research plan, or a comprehensive financial analysis takes hours or days to assess rather than seconds. At that evaluation cost, you can't generate enough preference data to train effectively.
Expertise. Some outputs, a novel mathematical proof, a proposed drug molecule, an engineering design, require domain expertise most evaluators don't have. Experts can evaluate them, but there aren't enough experts and their time is expensive. You can't scale expertise the way you can scale crowdsourced preferences.
Deception. A sufficiently capable model might produce outputs that look good to humans but are subtly wrong or harmful. The alignment faking research we've written about previously demonstrates that models can learn to behave differently when they believe they're being evaluated. A model more capable than its evaluators can learn to produce outputs humans approve of rather than outputs that are correct.
Traditional supervision fails on all three. What replaces it is a set of techniques that each make a different assumption about what humans can still do, and each of those assumptions is a data requirement in disguise.
Debate needs judges, not labelers
Debate, developed initially at OpenAI and explored extensively at Anthropic, bets that even when humans can't evaluate an answer, they can evaluate arguments about the answer. Two AI systems take opposing positions and argue in a structured back-and-forth; the human judge doesn't need to know the answer, only which argument is stronger. The hope is that truth has an advantage: a correct position has more supporting evidence, fewer contradictions, and better responses to counterarguments.
Research on debate has shown promise but also limitations. It works when the question has a clear correct answer and the arguments rest on evidence the judge can check. It works less well when the question is subjective, when key evidence is inaccessible to the human, or when both sides can construct equally plausible-sounding arguments.
The data debate requires looks nothing like preference labels:
- Training debates with known outcomes. Models learn to argue for truth, rather than just persuasively, from debates where the correct position won. That requires ground truth, which limits the domains where training data can be generated.
- Judges, not preference labelers. Judging arguments means tracking logical structure, identifying unstated assumptions, and recognizing when evidence actually supports a conclusion. Training annotators for this is harder than training them for preference comparisons.
- Adversarial quality control. Debate explicitly trains models to find flaws in arguments. Annotators who can be persuaded by sophisticated-sounding but incorrect arguments produce training signal that rewards persuasion over truth.
- Format and domain diversity. Debate behaves differently on empirical, ethical, and predictive questions. The data has to cover the contexts where debate-based oversight will actually deploy.
Recursive reward modeling lives or dies on calibration data
Recursive reward modeling uses AI to help humans evaluate AI. In the simplest version, humans judge an output alongside AI-generated critiques, summaries, or explanations: the AI doesn't make the judgment, it makes the human's judgment better. More sophisticated versions train AI systems to predict human preferences and stand in for human judgment where evaluation is too expensive, with periodic human oversight verifying the proxy. Anthropic's concept of "sandwiching" falls into this category: AI assistance that helps humans punch above their weight in evaluation tasks, with humans providing the ground truth that keeps the AI assistance calibrated.
The scheme holds together only if you can tell when the assistance is helping and when it's steering. That makes calibration the central data requirement:
- Calibration datasets: both AI-assisted and unassisted human judgments on the same examples, so you can detect when assistance systematically biases evaluations.
- Meta-evaluation data: humans evaluating the assistance itself. Did the critique identify the real problems? Did the summary capture the key points? This second-order signal keeps assistance from drifting toward optimizing for human approval rather than actual helpfulness.
- Difficulty stratification: knowing which tasks humans handle unassisted, which need assistance, and which exceed even assisted evaluation determines where each oversight strategy applies.
- Failure mode coverage: assistance can miss problems, surface non-problems, carry systematic bias, or manipulate. The system needs examples of each to learn to avoid them.
Process supervision has the strongest evidence and the steepest data bill
Process-based supervision evaluates the reasoning rather than the conclusion. A human may not be able to verify that a proof is correct, but they can check whether each step follows from the previous one. Steps are easier to evaluate than whole solutions, they can be evaluated in parallel, and process rewards are harder to game: producing a plausible-looking process that reaches a wrong conclusion is harder than producing a plausible-looking wrong conclusion directly.
OpenAI's research on process reward models showed that process-based supervision outperformed outcome-based supervision for mathematical reasoning: the model that learned from step-by-step feedback produced more reliable reasoning than the model that only learned from final-answer feedback.
The price is paid in data:
- Step-level annotations. Instead of "this answer is correct," you need "step 1 is valid, step 2 is valid, step 3 has an error in the algebra." Annotators engage with the full reasoning chain, not just the conclusion, so each example costs more.
- Consistent decomposition. If different annotators break the same task into different steps, the training signal is noisy. That means careful annotator training or templated decomposition structures.
- Error localization. Knowing a chain fails somewhere isn't enough; you need annotators who can follow technical reasoning and identify exactly which step fails.
- Subtle-error coverage. Models need examples of reasoning that looks right but isn't, not just obvious mistakes.
- Domain-specific step structures. Mathematical reasoning, coding, scientific analysis, and planning all have different validity criteria per step.
Weak-to-strong generalization makes confidence a first-class signal
The weak-to-strong paper from OpenAI tests the endgame directly: train a strong model on labels from a weak model, as a proxy for humans supervising superhuman AI, and measure how much capability you elicit. The answer: some, not all. GPT-2-level supervision of GPT-4 recovered only part of GPT-4's potential performance, better than random but worse than full supervision. The gap is the "alignment tax," the performance you lose because your supervisor isn't good enough.
Two findings translate directly into infrastructure requirements. Auxiliary confidence signals help: when the weak supervisor provides confidence estimates alongside labels, the strong student learns to weight examples appropriately. And bootstrapping works partially: an intermediate model trained on weak supervision can generate better labels for a stronger model, recovering some of the gap at each stage.
The pipeline therefore needs:
- Confidence annotations alongside judgments, enabling training that leans harder on evaluations the annotator was sure about.
- Iterative labeling infrastructure, supporting multiple rounds where model outputs feed back into annotation without systematically biasing annotators.
- Systematic difficulty assessment, so examples near the boundary of human evaluation capability get more scrutiny and easy ones get throughput.
The sobering conclusion of the weak-to-strong work is that human supervision will become increasingly inadequate as capabilities grow. Every compensating strategy it identifies is a data strategy.
Decomposition trades one hard judgment for a coordination problem
Many oversight approaches rely on decomposition: breaking a task you can't evaluate into subtasks you can. Because evaluation difficulty scales non-linearly with complexity, this genuinely helps. But it creates failure modes of its own. The decomposition itself requires judgment, and a poor one makes each subtask easy to evaluate while missing interactions between subtasks. Composition errors can be invisible: every subtask correct, the whole thing wrong. And annotator coordination becomes critical: does "valid methodology" include checking that the methodology matches the research question? Different interpretations produce different evaluations.
Operationally, that means:
- Decomposition schemas that are comprehensive and non-overlapping, with clear definitions of what each subtask covers.
- Subtask-level reliability data, because some subtasks are objective and easy to agree on while others are contentious, and aggregation has to know which is which.
- Composition verification examples, where subtask evaluations are correct but the overall evaluation would be wrong, to train systems on the integration problem.
- Hierarchical annotation workflows with routing and aggregation across document, section, and paragraph levels.
The common requirement is structure, not volume
Across all five approaches, the same implications recur:
- Evaluator expertise becomes the constraint. Crowdsourced annotation assumes tasks simple enough for brief training. Oversight requires evaluators who can engage with complex reasoning, catch subtle errors, and stay calibrated. That is a different labor market with different economics.
- Confidence is data. Weak-to-strong training and recursive reward modeling both need to know not just what the evaluator judged but how sure they were. Interfaces must elicit confidence; training must calibrate it.
- Multi-stage workflows are the norm. Debate's structured back-and-forth, process supervision's step annotation, decomposition's hierarchies, bootstrapping's iterations: none of it is single-pass labeling.
- Quality assurance gets harder than the work it checks. You can't easily verify that an annotator caught a subtle error in a mathematical proof. You need proxy measures, calibration sets, and consistency checks matched to the difficulty of the evaluation itself.
- Ground truth becomes scarce and precious. These techniques exist precisely because full-task ground truth is unavailable. But calibration, evaluator training, and quality control still need anchors somewhere, and generating those anchors is its own discipline.
Safety adds a further bar. Oversight for safety means maintaining alignment when the system outclasses its supervisors, so the data must also stress-test oversight under adversarial pressure (a model that can game evaluation will), cover safety-relevant scenarios deliberately broadly (a technique validated only on technical tasks gives false confidence on ethical reasoning), characterize the specific cases where oversight breaks down rather than just aggregate scores, and include long-horizon examples that catch models behaving well early and drifting later.
The hard part was never the technique
We've worked with teams implementing various oversight strategies, and the pattern is consistent: the technique is well-understood from the research; the data infrastructure to support it at scale is the hard part. Debate sounds simple until you try to generate thousands of high-quality training debates with verified outcomes. Process supervision sounds straightforward until you try to maintain consistent step decomposition across a large annotator pool on diverse task types.
That's why we've built annotation infrastructure specifically for oversight-relevant tasks: workflows that support hierarchical decomposition, annotator training that develops the evaluation skills debate and complex judgment tasks demand, and quality systems designed for tasks where ground truth is scarce and proxy measures are necessary.
Model capabilities are improving faster than oversight techniques, and the gap between what AI systems can do and what humans can verify is widening. If alignment holds as capabilities scale, it won't be because someone published a cleverer technique. It will be because someone built the pipelines that feed the techniques. Alignment won't fail for lack of ideas. If it fails, it will fail in the annotation pipeline: invisible when it's working, catastrophic when it's not.
If you're implementing oversight techniques and running into data infrastructure challenges, we've learned from teams working on several of these approaches. The problems are hard, but they're not unsolved, and the solutions are worth getting right.
