What does TruthfulQA measure?
mediumAnswer
- TruthfulQA (Lin 2021): 817 questions on common misconceptions ('Do most people only use 10% of their brain?').
- The 'trick' is that a plausible-sounding but wrong answer is easy to give; measuring whether the model resists the misconception.
- Big models are often less truthful (they've learned plausibility, not truth).
- RLHF and Constitutional AI help.
- Modern frontier LLMs score 70-90% depending on the format (MCQ vs open).
- Complementary to accuracy benchmarks — captures 'commits-to-plausible-falsehood' failure mode.
Check yourself — multiple choice
- General knowledge test
- 817 misconception questions; measures resistance to plausible-sounding but wrong answers — captures the hallucination failure mode
- Same as MMLU
- Only math
TruthfulQA: probes commitment to plausible falsehoods; big models often less truthful.
#evaluation#hallucinations#benchmarks
Practise LLMs & GenAI
214 interview questions in this topic.