'Verifier + refiner' RL pattern for reasoning.
hardAnswer
- (1) Sample many reasoning chains from model.
- (2) Verifier (external tool or LLM judge) scores each.
- (3) Best-of-N: keep highest scoring — cheap test-time scaling.
- (4) RL fine-tune: train model to produce chains verifier likes.
- (5) Iterative: use fine-tuned model to generate next round of chains, retrain.
- Foundation of o1, DeepSeek-R1, and many modern reasoning models.
- Test-time compute becomes tunable.
Check yourself — multiple choice
- Random
- Sample N chains → verifier scores → best-of-N test-time scaling → RL fine-tune to produce verifier-liked chains → iterate; foundation of o1 / DeepSeek-R1 modern reasoning
- Just sample
- Not real
Verify-and-refine: sample + verify + RL fine-tune + iterate; o1 / R1 pattern.
#llm#alignment
Practise Reinforcement Learning
214 interview questions in this topic.