How do you evaluate multimodal LLMs?
mediumAnswer
- (1) VQA benchmarks: VQAv2, GQA (visual question answering).
- (2) Complex reasoning: MMMU (multi-modal MMLU), MMBench, ScienceQA.
- (3) OCR / documents: DocVQA, ChartQA, InfographicVQA, TextVQA.
- (4) Fine-grained: RealWorldQA, HallusionBench (VLM hallucinations).
- (5) Video: Video-MME, EgoSchema.
- (6) Grounding: RefCOCO (predict box for 'the man in a red shirt').
- Each captures different capabilities.
- Combine with human eval on real production images.
Check yourself — multiple choice
- Only MMLU
- VQA + MMMU + DocVQA/ChartQA + HallusionBench + Video-MME + RefCOCO — multi-benchmark coverage
- Same as text
- Impossible
VLM eval: VQA / MMMU / DocVQA / HallusionBench / video / grounding — cover diverse skills.
#multimodal#evaluation#benchmarks
Practise LLMs & GenAI
214 interview questions in this topic.