HumanEval / MBPP / SWE-bench — how do they test code?
mediumAnswer
- HumanEval (Chen 2021): 164 hand-written Python problems, model writes a function that passes unit tests.
- Saturated at ~95% for frontier.
- MBPP (Austin 2021): 974 problems, similar format, slightly easier.
- Both single-function.
- SWE-bench (Jimenez 2023): real GitHub issues from popular Python repos — model must produce a PR-worthy patch.
- Much harder (~40% for top models); measures real-world code editing across a repo.
- Extended by SWE-bench Verified (human-verified subset) and Aider Polyglot for multi-language.
Check yourself — multiple choice
- All single-function
- HumanEval / MBPP: single-function unit tests (saturated); SWE-bench: real GitHub issues, repo-scale editing (~40%)
- SWE-bench = HumanEval
- None measure code
HumanEval / MBPP: functions; SWE-bench: real repo issues — much harder.
#evaluation#benchmarks
Practise LLMs & GenAI
214 interview questions in this topic.