PulseFlow Tecnologia
← Insights
·7 min read·framework-playbook·observability·mlflow·logs
PulseFlow Tecnologia

An observability playbook for agents

Agent observability isn't an API-latency dashboard. It's the ability to reconstruct, after the fact, what an agent saw, decided, called, and spent - for any specific execution, not just in aggregate. LangChain's State of AI Agents survey confirms this is already the most-cited control among companies with agents in production; what's missing in most cases is organizing it into layers, not just collecting logs of everything without structure.

Prompt and context logs: separate, not mixed

The final prompt that reaches the model and the context selected to compose that prompt are two different things, worth logging separately: the prompt shows what was asked; the context log shows why that specific piece of information was included (or excluded). Without that separation, debugging why an agent answered wrong turns into guesswork.

Tool calls and decisions: the trail that actually matters

Every tool call - which one, with what arguments, what it returned - needs its own log, alongside the decision that led to that call. An agent that answers well but calls the wrong tool is a problem that only shows up at this level of detail, never in the final response's log.

Cost and latency: per execution, not monthly aggregate

LangChain's survey reinforces this indirectly: execution quality is the top reported concern, but without cost and latency instrumentation per individual execution, there's no way to know if a specific problem is an exception or a pattern. MLflow recommends capturing the distribution at launch and monitoring statistical deviation - that's only possible with granular per-execution data.

Failures: recorded as data, not discarded as an exception

Every failure - tool error, timeout, response rejected by evaluation - needs to become structured data, not just a generic error log. Recurring failure patterns are the most direct signal that something in the agent's design needs to change.

Evals as a continuous layer, not an isolated checkpoint

The evaluation embedded in the flow that MLflow describes - "evaluation probes... in active agentic workflows," with results in a machine-readable audit trail - is what turns eval from a one-time checkpoint into a continuous observability layer. LangChain's survey shows offline evaluation (39.8%) is already more common than online evaluation (32.5%) among surveyed companies - but both should coexist, not compete.

Security as observability, not a separate module

Security events - out-of-scope access attempts, anomalous usage patterns - belong in the same observability layer as cost and latency, not a parallel system nobody checks day to day.

Human feedback: the data that closes the loop

When a human corrects, approves, or rejects an agent's action, that feedback needs to be captured as structured data, tied to the specific execution - that's what later lets you calibrate automated evaluation against real human judgment.

The playbook's ten layers

  1. Prompt logs - what was actually asked of the model
  2. Context logs - why that specific piece of information got in
  3. Tool calls - tool, arguments, return
  4. Decisions - the reasoning that led to each tool call
  5. Costs - per execution, not just aggregate
  6. Latency - per execution and per step
  7. Failures - structured, not discarded as exception
  8. Evals - continuous, embedded in the flow
  9. Security - same layer, not a parallel system
  10. Human feedback - captured and tied to the execution

Sources