What are Evals?

Teams shipping agents need to know whether a prompt, model, tool, or context change improved the product. Evals turn expected behavior into repeatable checks that can run during development and in CI.

accuracy_eval.py

from agno.agent import Agent
from agno.eval.accuracy import AccuracyEval
from agno.models.openai import OpenAIResponses
from agno.tools.calculator import CalculatorTools

agent = Agent(
    model=OpenAIResponses(id="gpt-5.4-mini"),
    tools=[CalculatorTools()],
)

evaluation = AccuracyEval(
    name="Calculator accuracy",
    model=OpenAIResponses(id="gpt-5.4-mini"),
    agent=agent,
    input="What is 10 factorial?",
    expected_output="3628800",
)

result = evaluation.run(print_results=True)
assert result is not None and result.avg_score >= 8

AccuracyEval runs the agent, then uses the evaluator model to compare its response with the expected output.

Choose an Eval

Question	Eval
Does the response match an expected answer?	Accuracy
Does the response meet product-specific quality criteria?	Agent as Judge
Did the agent or team call the expected tools with the expected arguments?	Reliability
How long does the code take and how much memory does it use?	Performance
Do many judge and reliability cases pass together?	Eval Suites

Accuracy and agent-as-judge checks use a model to evaluate output. Reliability checks inspect recorded tool calls and arguments. Performance checks execute a function repeatedly and report runtime and memory statistics.

Build a Regression Suite

Eval suites run multiple Case definitions against agents or teams. Each case can check custom criteria, expected tool calls, or both.

evals.py

import sys

from agno.agent import Agent
from agno.eval import Case, cli
from agno.models.openai import OpenAIResponses
from agno.tools.calculator import CalculatorTools

agent = Agent(
    id="calculator-agent",
    model=OpenAIResponses(id="gpt-5.4-mini"),
    tools=[CalculatorTools()],
    instructions="Use calculator tools for arithmetic.",
)

CASES = (
    Case(
        name="factorial_uses_calculator",
        agent=agent,
        input="What is 10 factorial?",
        tags=("smoke",),
        criteria="States that 10 factorial equals 3628800.",
        expected_tool_calls=("factorial",),
    ),
)

if __name__ == "__main__":
    sys.exit(cli(CASES))

python evals.py --tag smoke --json-output tmp/evals.json

The CLI returns a failing exit code when a selected case fails and can write a JSON report for CI artifacts. See Eval Suites for selectors, timeouts, judge modes, and programmatic execution.

What to Evaluate

Start with behavior that matters to the product:

Product requirement	Check
Support answers cite the correct policy	Accuracy against reviewed expected answers
Responses follow your tone and escalation rules	Agent-as-judge criteria
Refund requests call the approval tool	Reliability tool-call checks
Search stays within a latency budget	Performance thresholds tracked by your test harness

Keep cases focused on one behavior so failures point to a clear prompt, model, context, or tool change.

Next Steps

Accuracy

Compare responses with expected answers.

Agent as Judge

Score custom quality criteria.

Reliability

Check tool calls and arguments.

Performance

Measure runtime and memory usage.

Eval Suites

Run many judge and reliability cases in CI.

Get Started

Basics

Advanced

Production

Providers

Other

Additional Resources

What are Evals?

Choose an Eval

Build a Regression Suite

What to Evaluate

Next Steps

Accuracy

Agent as Judge

Reliability

Performance

Eval Suites

​Choose an Eval

​Build a Regression Suite

​What to Evaluate

​Next Steps

Accuracy

Agent as Judge

Reliability

Performance

Eval Suites

Choose an Eval

Build a Regression Suite

What to Evaluate

Next Steps