RLHF

Reinforcement Learning from Human Feedback — a training technique where human raters compare model outputs, teaching models to be more helpful and aligned.

Syntax

generative-ai

1. Pretrain LLM
2. Supervised fine-tune on demonstrations
3. Train reward model on human comparisons
4. Optimize LLM with PPO against reward model

Example

generative-ai

// RLHF is used by:
// ChatGPT, Claude, Gemini to:
// 1. Follow instructions better
// 2. Refuse harmful requests
// 3. Be more honest and helpful
//
// Example preference data:
// Prompt: "Explain quantum entanglement"
// Response A: [technical jargon] -> lower score
// Response B: [clear analogy] -> higher score
// Reward model learns these preferences