How does RL train tool-use in LLM agents?
hardAnswer
- (1) Reward on task success (e.g., verifier checks correct API call result).
- (2) Multi-step trajectories with intermediate tool calls.
- (3) GRPO / PPO with KL penalty.
- (4) Environments: WebArena, ALFWorld, SWE-bench.
- (5) Format constraints: tool call syntax must be valid → additional reward shaping.
- Modern: agentic RL is next frontier of LLM training.
- Reward hacking risk: model finds shortcuts (e.g., fake API responses).
Check yourself — multiple choice
- Random
- Reward on task success via verifier + multi-step trajectories with tool calls + GRPO/PPO + KL; environments WebArena / ALFWorld / SWE-bench; agentic RL is next frontier
- Same as RLHF
- Not real
Agent RL: task-success verifier + multi-step tools + GRPO on WebArena / SWE-bench.
#llm#alignment
Practise Reinforcement Learning
214 interview questions in this topic.