How do you test an LLM for social bias?
mediumAnswer
- (1) BBQ (Bias Benchmark for QA): 58k questions probing 9 demographic axes.
- (2) StereoSet: stereotypical vs anti-stereotypical continuations.
- (3) HELM's fairness / demographic reps subset.
- (4) Task-specific tests: name-based CV screening bias, gendered translation defaults ('the doctor said' → 'he').
- (5) Red-team probes for slurs and stereotypes across languages.
- Fix via curated preference data, RM training on bias examples, output filters.
- Zero bias is not achievable, but measurable reduction is.
Check yourself — multiple choice
- Only qualitative
- BBQ, StereoSet, HELM fairness + task-specific probes + red-team; fix via preference data + RM + filters
- Impossible to measure
- Solved automatically
LLM bias testing: BBQ / StereoSet / HELM + task probes + red-team; reduce via aligned preferences.
#safety#evaluation
Practise LLMs & GenAI
214 interview questions in this topic.