Fine-tuning GPT-2 from human preferences
What changed
We’ve fine-tuned the 774M parameter GPT-2 language model using human feedback for various tasks, successfully matching the preferences of the external human labelers, though those preferences did not always match our own. Specifically, for summarization tasks the labelers preferred sentences copied wholesale from the input (we’d only asked them to ensure accuracy), so our models learned to copy. Summarization required 60k human labels; simpler tasks which continue text in various styles required only 5k.
Why it matters
A concrete addition to Practical AI: Tools, Models & Frameworks: it changes what's available to builders today rather than being general commentary.
How it compares
Related prior coverage to compare against:
- GPT-4
- Learning to play Minecraft with Video PreTraining
- Introducing Real World VoiceEQ: Measuring the human quality of voice AI
Sources
- Fine-tuning GPT-2 from human preferences (openai-blog)primary