PulseFlow Tecnologia
← Insights
·7 min read·technical-guide·evals·mlflow·agent-testing
PulseFlow Tecnologia

How to apply evals to AI agents

Evaluating an AI agent isn't the same as evaluating a chatbot: the text response can be perfect and the agent can still have called the wrong tool, wasted unnecessary time and money, or taken an action no human would approve. This guide is grounded in MLflow's documented practices for production-ready agents and LangChain's State of AI Agents survey data on what companies already do - and what they still don't.

Step 1: define success criteria beyond "the answer looks right"

MLflow recommends specifically tracking "faithfulness" (factual accuracy), completeness, and sufficiency of the response - three distinct dimensions of "looks good." A response can be factually accurate and still incomplete, or complete and still insufficient for the task.

Step 2: establish ground truth before measuring drift

Capture the quality distribution at the moment the agent launches. Without that baseline, there's no way to know whether a quality drop is real or just noise - MLflow recommends setting statistical thresholds that trigger an alert when the deviation from that initial distribution is abnormal.

Step 3: test tool use, not just text

An agent that answers well but calls the wrong tool, or passes the wrong parameter, fails in a way text evaluation doesn't catch. Tool-use tests need to specifically verify: the right tool was called, with the right parameters, in the right order.

Step 4: security tests as their own category

The OWASP Agentic Skills Top 10 maps security risks specific to agent behavior layers - that's not the same as testing response quality. Security tests evaluate whether the agent resists manipulation attempts, respects scope boundaries, and fails safely when something goes wrong.

Step 5: regression on every change

Every change to a prompt, tool, or context is a candidate to break something that already worked. Regression tests running on every change catch this before production, not after.

Steps 6 & 7: cost and latency as evals, not a separate infra metric

Cost and latency shouldn't be measured only by the infrastructure team - they're part of the eval, because a correct response that costs 10x more or takes 10x longer might not be acceptable for the use case.

Step 8: traceability as part of the evaluation

An agent that decides correctly but leaves no trace of why it decided that way fails an audit, even while passing every other test. Evaluating traceability means checking whether the generated log is enough to reconstruct the decision later.

Step 9: evaluation embedded in the flow, with LLM-as-a-Judge for scale

The approach MLflow describes, based on NIST guidance, embeds "evaluation probes... in active agentic workflows," with results in "machine-readable audit trails." To avoid depending on human review of every response, MLflow's LLM-as-a-Judge approach automates most of the evaluation, reserving human review for cases the automated judgment itself flags as questionable.

Step 10: continuous monitoring, not a one-time evaluation

Eval isn't a step you do once before launch - it's a continuous process. LangChain's data shows tracing and observability tools are already the most-cited control among companies with agents in production, precisely because a one-time evaluation doesn't catch degradation that happens months after launch.

Sources