When OpenAI published their work on Reinforcement Learning from Human Feedback in 2022, it established what felt like the canonical approach to AI alignment: collect human preference data comparing model outputs, train a reward model on those preferences, then optimize the language model against that reward signal. RLHF became synonymous with alignment itself.
Three years later, that equivalence has broken down. Direct Preference Optimization eliminates the reward model. Reinforcement Learning from AI Feedback uses model-generated preferences instead of human ones. Constitutional AI embeds values through written principles rather than pairwise comparisons. And the research increasingly suggests that careful data selection matters more than data volume.
We believe the through-line is this: alignment data is shifting from a volume problem to an allocation problem. The methods that are winning don't eliminate human judgment; they route it, spending it only on the cases where it genuinely adds value. Teams building alignment data infrastructure should be building for that routing, not for annotation throughput.
RLHF's bottleneck was always the second stage
The standard RLHF pipeline has three stages. Supervised fine-tuning teaches a base model to follow instructions through prompt-response pairs, where a few thousand carefully crafted examples can beat larger noisy datasets. Reward model training requires comparison data: given a prompt and two or more responses, which does the human prefer? This typically takes tens of thousands of comparisons, and their quality directly determines the quality of the reward signal. The final reinforcement learning stage, typically Proximal Policy Optimization, needs no additional human data, but its behavior is entirely shaped by what the reward model learned.
The bottleneck is clearly the second stage. Collecting high-quality preference comparisons is expensive and slow: annotators read and evaluate multiple outputs per prompt, subtle distinctions between "slightly better" and "much better" require careful judgment, and as we discussed in our piece on alignment faking, the evaluation context itself can influence the feedback you get. Every method that followed is, in one way or another, an attack on this bottleneck.
DPO raises the quality bar on the same data
The insight behind DPO, introduced by Stanford researchers in 2023, is that you can skip the reward model entirely: a mathematical reformulation lets you update the language model directly from preference data. Training becomes simpler, more stable, and often faster, and the same preference dataset works for either RLHF or DPO.
That makes preference comparison data more valuable, not less. But it also raises the quality bar. Without a reward model to smooth over noise, DPO is more sensitive to inconsistent or low-quality comparisons. Annotator training, clear evaluation criteria, and robust quality assurance become more important, not less.
RLAIF moves human judgment upstream
RLAIF attacks the bottleneck differently: use a capable AI model to generate the preference labels. The results are hard to ignore. ICML 2024 papers demonstrated that RLAIF can achieve approximately 70% of the improvement over supervised fine-tuning that RLHF achieves, at roughly 100x lower cost in human annotation. For many applications, that tradeoff is compelling.
What's underappreciated is that RLAIF doesn't eliminate human judgment; it moves it upstream. Instead of labeling every comparison, humans design the evaluation criteria, craft the prompts for the AI judge, and validate that the AI's preferences are actually calibrated to human values. That is a different kind of human input, requiring prompt engineering and evaluation methodology rather than large-scale annotation. And the oversight is not optional: RLAIF amplifies whatever biases the evaluator model carries, so humans checking AI-generated preferences against human intuitions across diverse cases remain essential.
Constitutional AI trades annotation scale for principle quality
Anthropic's Constitutional AI rethinks the problem more radically: embed values through explicit principles, a "constitution," rather than learning them from examples. The model critiques and revises its own responses against those principles, creating supervised data without humans writing harmful content, then generates preference pairs from the same principles to drive reinforcement learning.
The core "data" here is the constitution itself: human-written principles like "please choose the response that is most supportive and encouraging of life, liberty, and personal security" or "please choose the response that is least racist and sexist." A few hundred carefully chosen principles might shape model behavior more than hundreds of thousands of preference comparisons. But crafting them is a different kind of labor: philosophical clarity about values, understanding of how principles translate to behavior, and empirical iteration on what works. Humans are still needed to evaluate whether the constitution produces the intended behavior, identify gaps, and validate interpretation. The work shifts from scale-intensive annotation to quality-intensive principle development.
The frontier is allocation: route hard cases to humans
The most interesting results come from hybrid approaches that use each data type where it's strongest. Research on RLTHF (Reinforcement Learning from Theory and Human Feedback) bootstraps alignment with theoretical principles, then refines with targeted human feedback, reaching full human-quality alignment with only 6-7% of the annotation effort of pure RLHF. The insight: most preference comparisons are "easy," where any reasonable approach agrees, and human annotation should concentrate on the hard cases where human judgment genuinely differentiates.
That implies a concrete data strategy: AI feedback handles routine cases, expert human feedback targets the difficult and ambiguous ones, and constitutional principles provide the underlying value framework. The infrastructure challenge is the routing itself: knowing which case is which, and building systems that can send each example to the right feedback mechanism.
Four consequences for data teams
First, preference comparison data remains foundational across methods. Whether you run RLHF, DPO, or validate RLAIF outputs, the ability to collect high-quality human preferences pays off regardless of which training method wins.
Second, the quality bar is rising. DPO's sensitivity to noise and hybrid methods' focus on hard cases both increase the value of careful judgment. Rushed annotation optimized for throughput is worth less than it used to be; deep expertise in evaluation methodology is worth more.
Third, flexibility is a competitive advantage. The "best" approach changes as new research lands. Infrastructure that supports pairwise preferences and scalar ratings, AI and human evaluation, and rapid iteration on criteria positions you to adapt.
Fourth, human judgment is being reallocated, not replaced. RLAIF and Constitutional AI reduce annotation volume but increase the sophistication required: prompt engineering for AI evaluators, principle development, validation of AI-generated preferences. The job is changing, not disappearing.
Pure RLHF, with humans labeling every comparison, is giving way to approaches that spend human judgment strategically, and that's good for alignment: the efficiency gains put it within reach of more organizations. But it raises the bar for infrastructure. The question is no longer "RLHF versus alternatives." It's "which combination of methods achieves your alignment goals under your constraints," and the teams that can answer it are the ones whose pipelines already know where human judgment matters most.
We work with teams navigating these tradeoffs across alignment methods. If you're thinking through your alignment data strategy and want to compare approaches, reach out.
