How do you evaluate an agent (tool-using LLM)?
hardAnswer
- (1) Task-level success rate on a set of goals with verifiable outcomes (booked flight, generated code that passes tests, delivered file).
- (2) Trace-level metrics: number of tool calls, tool-call accuracy, dead-ends.
- (3) Cost per successful task (tokens + tool costs).
- (4) Human eval on ambiguous / open-ended goals.
- (5) Regression suite of past failure cases.
- (6) Benchmarks: SWE-bench (coding agents), AgentBench, WebArena, VisualWebArena, OSWorld.
- Agents fail in complex ways — always sample real traces during monitoring.
Check yourself — multiple choice
- Only accuracy
- Task success rate + trace metrics + cost/success + regression suite + benchmarks (SWE-bench, AgentBench, WebArena)
- One number
- Impossible
Agent eval: task success + trace metrics + cost + regression suite + benchmarks.
#evaluation#agents
Practise LLMs & GenAI
214 interview questions in this topic.