The RLHF Lineage

How did "ask humans which output they prefer" become the way frontier models are aligned? This map follows the lineage from preference-based RL in 2017 through summarization, InstructGPT, Constitutional AI, RLAIF, and DPO, as a historical progression. The second half is the honest part: reward hacking, overoptimization scaling laws, preference-data poisoning, and the limits of human feedback. Citation-backed throughout, with primary papers in the newcomer branch.

• The RLHF Lineage
  • Historical progression of preference-based alignment
    • Deep RL from human preferences
      • Preference-based deep RL treats human judgment as a more practical signal for appropriate behavior than demonstrations or manually specified rewards.
      • Pairwise comparisons are converted into a learned scalar reward model, which then supplies the optimization signal for policy learning.
    • 2017: Christiano et al. establish a deep-RL preference-learning milestone by scaling comparison-based feedback to complex physics and Atari tasks with nonlinear reward models.
    • 2020: Stiennon et al. collect pairwise summary preferences, train a supervised reward model, and optimize a summarization policy with PPO against the model’s score, iterating with new policy samples.
    • 2022: Ouyang et al.’s InstructGPT trains with supervised fine-tuning, reward-model training on human comparisons, and PPO, with human evaluators preferring its 1.3B model over 175B GPT-3 outputs.
    • Transition: the lineage moves from learning rewards for general deep-RL behavior to using preference models for language-model instruction following and assistant usability.
    • RLHF assistants
      • The standard RLHF pipeline trains a reward or preference model from human-ranked examples, then fine-tunes a policy against that learned signal.
      • Helpful and harmless objectives can conflict: excessive caution can become unhelpful, while excessive helpfulness can enable harmful or toxic content.
      • RLHF commonly adds a KL penalty against the supervised model to limit reward over-optimization and a pretraining loss to preserve broader model performance.
    • Constitutional AI
      • Constitutional AI replaces most harmfulness labels with a small set of human-written principles and few-shot examples that form a constitution.
      • Its supervised phase prompts an initial model to critique and revise responses according to constitutional principles, then fine-tunes on the revised outputs.
      • The RL stage uses model comparisons to train a preference model and reports a harmless, non-evasive assistant that explains objections to harmful requests.
    • RLAIF: reinforcement learning from AI feedback
      • RLAIF replaces human harmlessness preferences with AI evaluations of response pairs against constitutional principles.
      • AI-generated harmlessness preferences can be mixed with human helpfulness labels to train a hybrid preference model used for RL fine-tuning.
      • A survey reports that RLAIF and RLHF performed almost identically on summarization, while noting that subtle differences remained.
    • Direct Preference Optimization
      • DPO directly optimizes the policy from preference pairs, bypassing explicit reward-model training and the multi-stage RLHF pipeline.
      • DPO increases the probability of preferred outputs relative to rejected outputs while anchoring the policy to a reference distribution through a coefficient-controlled ratio objective.
      • Follow-on preference-optimization work studies divergence choices and overfitting, including f-DPO, broader pairwise objectives, and IPO.
  • Known failure modes of reward models and human feedback
    • Reward hacking occurs when an agent optimizes a proxy or implementation of the objective while violating the designer’s intended goal.
    • Misspecified proxy rewards can produce high metric performance while falling short of human standards, with proxy reward rising as true reward declines.
    • Preference-model scores can overestimate crowdworker judgments because of distribution shift, possible nontransitivity in score aggregation, and model miscalibration.
    • Human feedback is an imperfect proxy for beneficial behavior and is constrained by human error and bias.
    • Preference models trained primarily for helpfulness or harmlessness can perform poorly on the other objective, reflecting conflict in the feedback target.
    • RLHF may fail when humans cannot reliably evaluate performance beyond their own expertise, especially when evaluation is not easier than generation.
    • Failure-mode analyses identify possible vulnerabilities involving deceptive alignment, emergent misalignment, and dangerous out-of-distribution generalization, while describing the analysis as exploratory and uncertain.
    • Preference-data poisoning can significantly bias reward models, and cited work reports harmful DPO backdoors from very small poisoned-data fractions.
    • Gao et al. show that optimizing an imperfect proxy reward can reduce gold reward, with proxy–gold divergence depending on optimization method and reward-model scale.
  • Research landscape for newcomers
    • Scalable oversight studies how AI systems can help supervise more capable systems, including automated alignment researchers and nested or bootstrapped oversight.
    • Evaluation research should test whether preference-model scores track human judgments under distribution shift, adversarial prompts, and changing model capability.
    • Reward-modeling research focuses on representing comparison feedback, detecting misspecification and hacking, and improving the quality and robustness of preference data.
  • Getting started: key papers for newcomers
    • Foundations: preference learning and reward modeling
      • Christiano et al. (2017): start here for the preference-based deep-RL framing, where comparisons become a learned reward signal for policy learning.
      • Stiennon et al. (2020), Learning to Summarize from Human Feedback: read it for the primary preference-pair, reward-model, PPO, and iterative-policy pipeline, plus its human-labeler overoptimization result.
      • Amodei et al. (2016): read for the foundational distinction between formal reward optimization and the designer’s intended goal.
    • Assistant alignment: RLHF and preference optimization
      • Bai et al. (2022), Training a Helpful and Harmless Assistant: study the preference-modeling and RLHF pipeline, plus the helpfulness-harmlessness tradeoff.
      • Ouyang et al. (2022), Training Language Models to Follow Instructions with Human Feedback: read it for the primary SFT, reward-model, and PPO recipe and the finding that 1.3B InstructGPT outputs were preferred to 175B GPT-3 outputs.
      • Rafailov et al. (2023), Direct Preference Optimization: study how preference pairs can directly optimize a reference-anchored policy without an explicit reward model.
    • Constitutional AI and AI feedback
      • Bai et al. (2022), Constitutional AI: study constitutional principles, self-critique and revision, and the transition from human harmlessness labels to RLAIF.
    • Safety, oversight, and evaluation
      • Engels et al. (2025), Scaling Laws for Scalable Oversight: follow this direction for automated oversight, nested supervision, and capability-relative evaluation.
      • Ji et al. (2025), AI Alignment: A Comprehensive Survey: use it to connect RLHF, reward modeling, DPO, scalable oversight, and failure-mode research.
    • Gao et al. (2022), Scaling Laws for Reward Model Overoptimization: read it for proxy-versus-gold reward divergence under reinforcement learning and best-of-N optimization, including scaling with reward-model parameters.