LLM agent benchmarks — what tests capability?
mediumAnswer
- (1) WebArena / VisualWebArena: browse web + execute tasks.
- (2) ALFWorld: navigate textual household environment.
- (3) SWE-bench: fix real GitHub issues.
- (4) AgentBench: comprehensive tool-use.
- (5) OSWorld: real OS interaction.
- (6) τ-bench: customer-service dialog tasks.
- Rule: test in realistic settings (not just multiple choice); track success rate + cost per task + safety violations.
- Modern agent RL trained on some of these.
Check yourself — multiple choice
- Random
- WebArena / VisualWebArena (web) + ALFWorld (household) + SWE-bench (GitHub) + AgentBench (tools) + OSWorld (OS) + τ-bench (dialog) — realistic settings + track success + cost + safety
- Same as MMLU
- Not real
Agent bench: WebArena / ALFWorld / SWE-bench / AgentBench / OSWorld / τ-bench.
#llm#evaluation
Practise Reinforcement Learning
214 interview questions in this topic.
Related questions
- What is RLHF and how does it fit into training an LLM?
- What is off-policy evaluation and why is it hard?
- DPO — Direct Preference Optimization instead of RLHF PPO.
- GRPO — Group Relative Policy Optimization (DeepSeekMath, R1).
- RLHF pipeline — the three stages in detail.
- Bradley-Terry model in reward modeling.