What LLM observability tools do you deploy?
mediumAnswer
- (1) LLM-specific tracing: Langfuse, LangSmith, Helicone, Braintrust — capture prompts, retrieved contexts, tool calls, outputs, latency, cost per trace.
- (2) Feedback: thumbs-up/down + free-text feedback tied to traces.
- (3) Prompt / model version tracking.
- (4) Automatic eval on production samples (LLM-as-judge).
- (5) Regression detection: alert when metrics drift after model or prompt change.
- (6) PII redaction before logging.
- Combine with metric stacks (Prometheus, Datadog) for GPU / latency.
Check yourself — multiple choice
- Only latency logs
- LLM-tracing (Langfuse / LangSmith) + user feedback + prompt / model versions + auto-eval + regression alerts + PII redaction
- No tools exist
- Only Datadog
LLM observability: trace stacks + feedback + versioning + auto-eval + regression + PII.
#production#serving#evaluation
Practise LLMs & GenAI
214 interview questions in this topic.