← Home
TWIML · May 7, 2026 · 53m

How to Find the Agent Failures Your Evals Miss

Scott Clark
Show Notes

Scott Clark, co-founder and CEO of Distributional, discusses how teams can reliably operate and improve complex LLM systems and agents in production. The episode explores a Maslow's hierarchy of observability framework, real-world production failures that standard evals miss (like lazy tool-use hallucinations), and how vector fingerprinting of traces enables clustering to uncover emergent behaviors. Clark explains how online analytics feed a data flywheel to generate evals, guardrails, and training data, and why adaptive approaches are essential for non-stationary models.

This summary was generated from show notes and public descriptions, not from a full transcript review. Details may contain inaccuracies.

Novel

•
Lazy Tool-Use Hallucinations
Agents can hallucinate tool-use by claiming to call tools without actually executing them — a failure pattern that standard evals miss entirely.
ClaudeChatGPTGemini

Highlights

•
Maslow's Hierarchy of Observability
Scott introduces a three-tier observability framework: telemetry (logging), monitoring (known signal tracking), and online analytics (discovering unknown unknowns).
ClaudeChatGPTGemini
•
Vector Fingerprinting for Failure Clustering
Mapping execution traces into vector fingerprints enables unsupervised clustering to discover patterns across similar failures without predefined categories.
ClaudeChatGPTGemini
•
The Analytics Data Flywheel
Production analytics feed a continuous loop: observability uncovers failures → failures generate new evals → evals create guardrails and training data → better models prevent future failures.
ClaudeChatGPTGemini
•
Online, Adaptive Observability for Non-Stationary Systems
LLM systems are non-stationary — behavior drifts over time with model updates, user patterns, and emerging edge cases — requiring adaptive observability that evolves rather than static monitoring.
ClaudeChatGPTGemini

Misc

✧"Lazy" tool-use hallucinations — agents claiming to use tools without actually calling them — are a production failure pattern that standard evals completely miss
✧Vector fingerprints of traces can cluster similar failures, revealing emergent behaviors that weren't explicitly tested
✧The data flywheel: production analytics → failure discovery → new evals → guardrails → training data → better models
Was this useful?