How well-calibrated are LLM confidences and how do you fix them?
hardAnswer
- Base LLMs are decently calibrated (token probabilities match observed accuracy in-distribution).
- RLHF-tuned LLMs are overconfident — asked 'how sure are you?' they say 90% but are right 60% of the time.
- Fixes: (1) verbalized confidence prompts + calibration on a labeled dev set; (2) sampling-based confidence (fraction of K samples that agree); (3) explicit calibration training (e.g., True/False + Confidence-tuning); (4) ensembling.
- Calibration matters for downstream decisions: threshold on the LLM's confidence before deciding to escalate to a human.
Check yourself — multiple choice
- LLMs are perfectly calibrated
- Base LLMs OK, RLHF-tuned are overconfident; use verbalized confidence + calibration on dev set / sampling agreement / ensembling
- Only classification
- Impossible to calibrate
LLM calibration: RLHF makes overconfident; verbal + sampling + calibration fix.
#reliability#evaluation#alignment
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- What is sycophancy in LLMs and how do you reduce it?
- What is 'honesty' as an alignment target and how do you train for it?
- How reliable is LLM-as-a-judge and how do you validate it?
- How do you detect hallucinations in production?
- Why do LLMs hallucinate and how do you reduce hallucinations?
- How do you evaluate an LLM system?