How do reward models scale with size?
hardAnswer
- Larger RMs (closer in size to policy) give better preferences: better calibration, less easy-to-exploit gaps.
- But: too small → doesn't understand nuance; too large → expensive per training step.
- Modern: RM around 1/3 of policy size (7B RM for 22B policy).
- Ensemble of RMs improves robustness against hacking.
- Some frontier labs use LLM-as-judge (whole LLM) instead of scalar-output RM head.
Check yourself — multiple choice
- No scaling
- Larger RMs give better preferences / less exploit gaps but expensive; modern RM ≈ 1/3 policy size (7B for 22B); ensembles + LLM-as-judge for robustness against hacking
- Random
- Same size always
RM scaling: bigger better; RM ≈ 1/3 policy size; ensembles + LLM-judge.
#llm#alignment
Practise Reinforcement Learning
214 interview questions in this topic.