RLHF-specific ops challenges.
hardAnswer
- (1) Multiple models in memory: policy + reference + reward model + critic — 3-4x memory pressure.
- (2) Rollout generation is bottleneck: use vLLM for rollouts + separate training.
- (3) Reward hacking detection: monitor KL divergence + downstream metric — if KL grows fast but downstream doesn't improve → likely hacking.
- (4) Sample diversity monitoring.
- (5) Preference data collection pipeline: annotators, quality checks, active sampling.
- (6) RM scaling limits — retrain periodically.
- (7) Frameworks: TRL, OpenRLHF, DeepSpeed-Chat, LMSYS trlX.
- Complex + fragile.
Check yourself — multiple choice
- Just PPO
- Multi-model memory 3-4x + rollout vLLM separate from train + reward hacking KL vs downstream + diversity + preference pipeline + RM retrain + TRL/OpenRLHF/DeepSpeed-Chat/trlX; complex + fragile
- Random
- Not real
RLHF ops: multi-model memory + vLLM rollout + KL vs downstream + preference pipeline.
#llmops#reproducibility
Practise MLOps & Data Quality
215 interview questions in this topic.