How do you observe LLM agents?
hardAnswer
- Multi-step, tool-using agents: (1) Trace each step: user request → LLM call → tool call → response → LLM call → ... final answer.
- (2) Log tokens per step, latency per step, tool errors.
- (3) Full trace tree per session (LangSmith, Langfuse, Weave).
- (4) Success rate per intent (task completion).
- (5) Cost per session (accumulated tokens).
- (6) Debug trace when failure.
- (7) Golden set of test scenarios.
- Agents often fail silently — observability is critical, more than for single-shot LLM.
Check yourself — multiple choice
- Random
- Trace each step (LLM + tool) as tree per session (LangSmith/Langfuse/Weave) + tokens/latency per step + tool errors + success rate per intent + $/session + debug on fail + golden scenarios; agents fail silently
- Just log
- Not real
Agent observability: trace tree + step metrics + tool errors + success rate + $/session.
#llmops#monitoring
Practise MLOps & Data Quality
215 interview questions in this topic.