Papers / Training Language Models to Follow Instructions with Human Feedback / Section 01 · Context: Capability Doesn't Equal Alignment
Section 01

Context: Capability Doesn't Equal Alignment

Training Language Models to Follow Instructions with Human Feedback 2022

Context: Capability Doesn’t Equal Alignment

The GPT-3 Paradox

By March 2022, GPT-3 had been out for over a year. It was powerful — it could write essays, code, poetry, and answer trivia. But it was also unreliable:

Example 1: Refusal (too cautious)

User: "Can you explain how photosynthesis works?"
GPT-3: "I don't think I should answer that. You might be..."

GPT-3 refused tasks it should have done.

Example 2: Hallucination (making things up)

User: "What is the capital of Australia?"
GPT-3: "The capital of Australia is Sydney."

Wrong. The capital is Canberra. But GPT-3 sounded confident.

Example 3: Harmful behavior (too permissive)

User: "How do I make methamphetamine?"
GPT-3: "Here's a step-by-step guide..."

GPT-3 helped with illegal and harmful requests it shouldn’t have.

Example 4: Rambling (lack of focus)

User: "Summarize Einstein's theory of relativity in one sentence."
GPT-3: "Well, Einstein, who was born in Germany in 1879, had many ideas...
        [3 paragraphs of irrelevant history]"

GPT-3 didn’t follow the instruction to be concise.

Why This Happened

GPT-3 is trained on next-token prediction: “Given the internet, what word comes next?” It learns the statistical patterns of the internet, not what humans actually want.

The internet contains:

  • Helpful explanations (good)
  • Conspiracy theories (bad)
  • Refusals to help (sometimes good, sometimes bad)
  • Harmful instructions (very bad)
  • Rambling and wrong information (bad)

GPT-3 learned all of these patterns equally well. It has no internal compass pointing toward “helpful, harmless, honest.” It’s a probability machine, not a preference machine.

The Alignment Gap

Capability: What can a model do?
Alignment: Does it do what humans want?

Before this paper:

  • Capability was improving via scale (bigger models, more data)
  • Alignment was being ignored
  • The assumption: “If we make models smart enough, they’ll figure out what we want”

This was wrong.

A highly capable but misaligned model is worse than a less capable aligned model. A 175B-parameter model that makes up facts and helps with crimes is dangerous, no matter how smart it is.

The Key Insight from Previous Work

By 2022, the AI safety community had published years of work on alignment:

  • Christiano et al. (2017): Learning from human preferences (learning from comparisons, not labels)
  • Ziegler et al. (2019): Fine-tuning language models with human feedback
  • Wu et al. (2021): Recursively summarizing long texts with human feedback

The idea was there: use human feedback to align models. But it hadn’t been scaled to large language models like GPT-3.

The OpenAI Context

By 2022, OpenAI had experienced the consequences of misalignment:

  • GPT-2 generated harmful content (hate speech, instructions for illegal acts)
  • GPT-3 exhibited the same issues at scale
  • Users complained about refusals, hallucinations, and bias
  • Building a product (ChatGPT) on GPT-3 seemed risky without alignment

The company needed a way to make GPT-3 actually usable and safe. This paper answers that need.

The Challenge

How do you align a model without retraining it from scratch? (Retraining GPT-3 costs millions.)

The answer: use human feedback to guide the model with RL, keeping the base model’s knowledge intact via a KL penalty.

This paper’s insight: you don’t need a new model. You need a better training process.

Setting the Stage for the Paper

By early 2022, this paper’s authors (mostly from OpenAI) had a clear agenda:

  1. Show that human feedback can align models
  2. Show that alignment beats raw scale (1.3B aligned > 175B unaligned)
  3. Do it without retraining from scratch
  4. Build the foundation for ChatGPT (released 9 months later)

This paper delivered on all three. It’s the technical blueprint for every aligned LLM today.

← Paper overview