What are the biases of LLM-as-judge?
hardAnswer
- (1) Position bias: prefers first / second response systematically → randomize order + swap.
- (2) Verbosity bias: prefers longer answers → normalize length.
- (3) Self-preference: judge favors output style similar to own model → use different judge model.
- (4) Familiarity bias: agrees with popular / common answers.
- (5) Rubric interpretation drift over runs.
- (6) Correlation with actual quality < 0.9 typically.
- Mitigate: multi-judge ensemble, human calibration, position swap, rubric with examples.
- Track judge-vs-human on golden set.
Check yourself — multiple choice
- None
- Position (prefer 1st/2nd) + verbosity (longer) + self-preference (similar-style) + familiarity (popular) + rubric drift; mitigate via swap + normalize + multi-judge + calibrate on human golden; track judge-vs-human correlation
- Random
- Not real
Judge biases: position + verbosity + self-preference + familiarity; calibrate.
#llmops#monitoring
Practise MLOps & Data Quality
215 interview questions in this topic.