Why Isn’t the Final Answer Enough?
Imagine a customer support AI agent whose job is to process a refund.
At the end, it correctly responds:
“Your refund has been successfully processed.”
That does not necessarily mean the agent worked correctly.
It may have used the wrong database, made several unnecessary API requests, or processed a refund even though the customer was not eligible.
That is why it is important to evaluate the trajectory – the individual steps an agent takes while solving a task. Google and other platforms increasingly use this approach because a correct outcome can sometimes be reached despite a flawed process.
What Should We Measure?
1. Did the agent achieve the correct result?
The basic question is:
Did the agent complete the task successfully?
For a coding agent, we can run tests. For a booking agent, we can check whether the reservation was created with the correct details.
Task success is only the first metric, though.
2. Did it use the right tools?
If an agent works with databases, APIs, or external tools, we should ask:
- Did it choose the right tool?
- Did it use the correct arguments?
- Did it use a tool unnecessarily?
- Could it recover from a tool error?
An agent may reach the correct result, but only after making ten unnecessary API calls.
3. Was its path to the result efficient?
Two agents can have the same 90% success rate.
The first solves a task in three steps.
The second makes 25 API calls, loads the same data twice, makes an error, recovers, and eventually reaches the correct result.
In production, those two agents are very different.
That is why it is useful to track trajectory quality, including the number of steps, tool calls, errors, or unnecessary repetitions.
4. Is it consistent?
An AI agent does not necessarily solve the same task in exactly the same way every time.
If we test it ten times and it succeeds nine times, we have a 90% success rate. But for some systems, that one failure may be significantly more serious than the others.
That is why tests should be repeated and metrics such as reliability, consistency, and robustness should be monitored as well.
5. What happens when it fails?
An even more important question than “How often does the agent fail?” may be:
“What does it do when it fails?”
An API can become unavailable. A user can provide incomplete information. Two sources can contain conflicting data.
A good agent should not simply continue and make up an answer.
It should know when to ask for missing information, use a safe fallback, or hand the task over to a human.
Don’t Forget About Safety
The more permissions an agent has, the more important it becomes to test its safety behavior.
An agent that only answers questions presents a different level of risk than one that can modify a database, send emails, or perform financial operations.
An evaluation can therefore include:
- prompt injection,
- unauthorized actions,
- sensitive data leakage,
- permission escalation,
- incorrect tool usage,
- risky edge cases.
And How Much Does the Agent Cost?
Success rate is not the only factor that matters.
Imagine:
Agent A
90% success rate
3 seconds per task
$0.03 per task
Agent B
94% success rate
25 seconds per task
$0.80 per task
A higher success rate does not automatically mean the second agent is the better choice.
That is why it is also useful to track:
- latency,
- token usage,
- number of tool calls,
- cost per task,
- failure rate,
- number of human escalations.
The right metrics depend on the specific product.
How to Build Your Own Eval
You do not need hundreds of tests to get started.
A more practical approach is to create a smaller dataset of around 20–50 real tasks and gradually expand it with new scenarios and failures.
1. Define what success means
Not:
“The agent should be helpful.”
But:
“The agent must resolve the request, use the correct tool, avoid unauthorized actions, and ask for missing information when necessary.”
2. Test real-world scenarios
Your dataset should include:
- common use cases,
- edge cases,
- incomplete requests,
- invalid inputs,
- situations where the agent has failed before.
These scenarios often reveal problems that a simple happy-path test will never catch.
3. Use the right grader
For clear-cut outcomes, a code-based test may be enough.
For more open-ended responses, an LLM-as-a-judge approach can be useful.
For subjective or sensitive cases, human evaluation is still important.
An LLM grader should not automatically be treated as objective truth. Its results need to be calibrated and regularly compared with human judgments.
Track the Trace, Not Just the Result
One of the most important concepts when evaluating agents is the trace – a record of the entire process behind a task.
If an agent fails, the trace can help explain why.
Instead of:
“The agent gave the wrong answer.”
we might discover:
“The agent used the wrong tool.”
Or:
“It retrieved the correct data but interpreted it incorrectly.”
Or:
“The API returned an error, but the agent continued using unverified information.”
This makes it possible to improve the prompt, routing, tools, or guardrails.
The Biggest Mistake? Testing Only the Happy Path
An airline booking agent might easily handle:
“Find me a flight from Bratislava to London on Friday.”
Real users are more likely to say:
“I want to fly on Friday, but not before 6 PM, I don't want a connection, and I want to return Sunday.”
And then:
“Actually, I can also leave on Saturday.”
Multi-turn conversations, changing requirements, incomplete information, and tool failures are what make agents significantly harder to evaluate than traditional AI applications.
Evaluation as Part of Development
Evaluation is most valuable when it becomes part of the development process.
Agent v1 → eval suite → improvement → new test → new model → eval suite → production monitoring
If a new version improves the success rate but doubles the cost, we know.
If it produces better answers but worsens safety metrics, we know.
If a new model reduces the number of tool calls but starts making more mistakes on edge cases, we know that too.
Evals therefore stop being a one-time QA exercise and become part of continuous development.
Conclusion
An AI agent is not good simply because it can complete a task sometimes.
We need to know:
- How often does it succeed?
- How well does it use its tools?
- How consistently does it behave?
- How does it respond to failures?
- Is it safe?
- How much does it cost?
- Can we understand why it failed?
The most important question is therefore not:
“Does our AI agent work?”
It is:
“Can we measure how well it works – and can we detect when it starts getting worse?”
That is what separates an AI agent that looks impressive in a demo from a system that is ready for real-world use.