What biases affect LLM-as-a-judge evaluation, and how do you control them?

hard

Answer

$Position bias: the judge favours whichever answer comes first, so swap the order and average, or run both orders and discard disagreements.$
$Verbosity bias: longer answers score higher regardless of quality, so control for length or instruct the judge to ignore it.$
$Self-preference: a model rates its own family's outputs higher, so avoid judging a model with itself.$
$Sycophancy toward assertive phrasing.$
$Controls that work: a concrete rubric instead of 'rate 1-5', few-shot examples of each score, forcing a short justification before the score, and validating the judge against human labels on a sample before trusting it at scale.$

Check yourself — multiple choice

Judges are unbiased
Position, verbosity, self-preference and sycophancy biases — control with order swapping, rubrics, justifications and validation against human labels
Only temperature matters
Judges cannot be used at all

Known judge biases need order randomization, rubrics and human validation.

#evaluation#benchmarks

214 interview questions in this topic.