How do you evaluate an agent's tool-use accuracy?

medium

Answer

$Per-step: (1) tool selection correct?$
$(2) arguments valid + well-formed?$
$(3) result correctly interpreted?$
$Per-trace: (1) task success rate (verified outcome); (2) number of steps to success; (3) cost per success; (4) recovery from tool errors.$
$Ground truth: hand-labeled successful traces or verified outcomes (e.g., 'the flight was actually booked').$
$Benchmarks: AgentBench, ToolBench, ToolEval, τ-bench, MLE-bench.$
$Analyze failure modes with clustered traces.$

Check yourself — multiple choice

Only final answer
Per-step: tool choice + args + result interpretation; per-trace: success rate + steps + cost + error recovery; benchmarks: AgentBench, τ-bench
One number
Impossible to evaluate

Agent eval: per-step tool metrics + per-trace success + cost + benchmarks.

#agents#evaluation

214 interview questions in this topic.