What are Evals?
Measure agent and team quality across expected answers, custom criteria, tool behavior, and performance.
Teams shipping agents need to know whether a prompt, model, tool, or context change improved the product. Evals turn expected behavior into repeatable checks that can run during development and in CI.
Before running these examples, install the dependencies in your Python environment and set your OpenAI key:
uv pip install -U agno openai
export OPENAI_API_KEY="your-api-key"On PowerShell, use $Env:OPENAI_API_KEY="your-api-key".
from agno.agent import Agent
from agno.eval.accuracy import AccuracyEval
from agno.models.openai import OpenAIResponses
from agno.tools.calculator import CalculatorTools
agent = Agent(
model=OpenAIResponses(id="gpt-5.4-mini"),
tools=[CalculatorTools()],
)
evaluation = AccuracyEval(
name="Calculator accuracy",
model=OpenAIResponses(id="gpt-5.4-mini"),
agent=agent,
input="What is 10 factorial?",
expected_output="3628800",
)
result = evaluation.run(print_results=True)
assert result is not None and result.avg_score >= 8AccuracyEval runs the agent, then uses the evaluator model to compare its response with the expected output.
Choose an Eval
| Question | Eval |
|---|---|
| Does the response match an expected answer? | Accuracy |
| Does the response meet product-specific quality criteria? | Agent as Judge |
| Did the agent or team call the expected tools with the expected arguments? | Reliability |
| How long does the code take and how much memory does it use? | Performance |
| Do judge, reliability, and scorer checks pass together? | Eval Suites |
Accuracy and agent-as-judge checks use a model to evaluate output. Reliability checks inspect recorded tool calls and arguments. Performance checks execute a function repeatedly and report runtime and memory statistics.
Build a Regression Suite
Eval suites run multiple Case definitions against agents or teams. Each case can combine custom judge criteria, expected tool executions, and a scorer implemented in code.
import sys
from agno.agent import Agent
from agno.eval import Case, cli
from agno.models.openai import OpenAIResponses
from agno.tools.calculator import CalculatorTools
agent = Agent(
id="calculator-agent",
model=OpenAIResponses(id="gpt-5.4-mini"),
tools=[CalculatorTools()],
instructions="Use calculator tools for arithmetic.",
)
CASES = (
Case(
name="factorial_uses_calculator",
agent=agent,
input="What is 10 factorial?",
tags=("smoke",),
criteria="States that 10 factorial equals 3628800.",
expected_tool_calls=("factorial",),
),
)
if __name__ == "__main__":
sys.exit(cli(CASES))python evals.py --tag smoke --json-output tmp/evals.jsonThe CLI returns a failing exit code when a selected case fails and can write a JSON report for CI artifacts. See Eval Suites for selectors, timeouts, judge modes, and programmatic execution.
What to Evaluate
Start with behavior that matters to the product:
| Product requirement | Check |
|---|---|
| Support answers cite the correct policy | Accuracy against reviewed expected answers |
| Responses follow your tone and escalation rules | Agent-as-judge criteria |
| Refund requests call the approval tool | Reliability tool-call checks |
| Search stays within a latency budget | Performance thresholds tracked by your test harness |
Keep cases focused on one behavior so failures point to a clear prompt, model, context, or tool change.