Building reliable AI agents by starting with evaluation

Recent work shows that building reliable AI agents requires a paradigm shift in software development towards agent evaluation¹²³. Automated unit and integration tests are written to guarantee that software performs as expected and to guard against feature regressions, for AI agents tests can no longer be written to cover all possible cases. The LLMs that underlie all agent interactions are non-linear neural networks that can respond in unpredictable ways, ways that are potentially different each time. It therefore follows that AI agents built from a set of prompts combined with a set of tools can, whilst generally fulfilling the required task successfully, also potentially fail in surprising and unpredictable ways. An LLM handed a bunch of tools and a job to do must be constantly monitored and assessed in production to identify unknown failure cases. It must be constantly interrogated:

• Are the correct tools being used?
• Does a new foundation model release result in an increase in expensive API calls?
• Is the agent safe?
• Does it leak the personal data of other customers?

The same evaluations that guard production performance and reliability can also provide an important feedback signal during development of the agent. For each change that enters the codebase we can quantify whether the change improves or degrades performance. As production traces are collected and used to measure system performance during testing, the behaviour of the agent can be honed using theses metrics. In this new way of working with AI, developers are moving from writing code to steering complex systems. Complex systems fail in ways that are only visible in the aggregate behaviour of the whole, and safety should be treated as an emergent property of the system.

Creating a basic tool use performance metric with DeepEval

Monitoring, logging, and evaluation provide the data to answer these questions. Evaluation is a core component of building safe and performant agents and combined with comprehensive tracing forms the basis of compliance with legal requirements.

DeepEval is one of the libraries that can be used to calculate performance metrics. To demonstrate writing evaluations I have created a sample project. This comprises a simple agent which can choose between two tools: one for fetching the weather at a given location, the other to retrieve an article summary from Wikipedia. We will see that the choice of foundation model can directly affect agent performance.

To probe the agent behaviour we need to write an LLMTestCase and apply a ToolCorrectnessMetric. The ToolCorrectnessMetric works by combining a deterministic score (the proportion of tools calls that were correct) with an LLM evaluation. The LLM evaluation takes the list of available tools and determines if the tools that were called were the most appropriate for the given context. The value returned is the lowest of the two numbers.

An LLMTestCase is instantiated with the input that was sent to the LLM, the output that it generated, an array of tools that were used by the LLM, and the tool that we expected to be used. This is all very similar to traditional automated testing in software development; an expected result, what should happen, and the real result that did happen. If the real output matches that expected then we can say that the test passed, the agent was successful. The ToolCorrectnessMetric is an LLM-as-judge method.

Running a DeepEval test suite requires a list of LLMTestCase objects and list of metrics that should be applied to them.

This opens up a lot of options for directing the development of the agent. The tool call success rate - all the correct tools called in the correct order with the correct parameters - helps us to see how each small change to the parts of the system affect the overall performance: system prompts, function definitions, foundation model choice etc. can each be tweaked independently.

Furthermore, the aggregated log data from production systems can be used to generate a performance signal to help steer the development of the agent. The agent can be improved in feedback loops, becoming increasingly better at the required task. In the sample codebase switching the base model from Gemini 3.5 to Ministral 3B results in a fall of tool calling performance to 75% from 100%.

Concluding remarks

Foregrounding evaluation during the agent development process means that we can immediately see performance metrics when log data is available from production systems. This has the compounding effect that we can iterate the agent with prompt and function definition improvements and know that the agent really is improving, whilst also ensuring that it is not failing in new ways that do not occur during testing. Developing agents in tandem with evaluations will ensure not only legal compliance but also guarantee, safe and reliable agentic AI.

References