> ## Documentation Index
> Fetch the complete documentation index at: https://docs.agno.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Agent Evaluation

> Catch regressions in response quality, tool use, latency, and memory.

Changes to models, instructions, tools, and knowledge can introduce regressions. Agno evals turn response criteria and expected tool use into executable cases. Run them during development, gate CI with their exit code, and evaluate selected production outputs through hooks.

```python evals.py theme={null}
import sys

from agno.agent import Agent
from agno.eval import Case, cli
from agno.tools.calculator import CalculatorTools

calculator = Agent(
    id="calculator",
    model="openai:gpt-5.5",
    tools=[CalculatorTools()],
    instructions="Use the calculator tools for every calculation.",
)

CASES = (
    Case(
        name="factorial_uses_calculator",
        agent=calculator,
        input="What is 10 factorial?",
        criteria="States that 10 factorial equals 3,628,800.",
        expected_tool_calls=("factorial",),
    ),
)

if __name__ == "__main__":
    sys.exit(cli(CASES))
```

Create a virtual environment, install the OpenAI integration, and set `OPENAI_API_KEY` before running the suite:

```bash theme={null}
uv venv --python 3.12
uv pip install -U "agno[openai]"
```

```bash theme={null}
uv run python evals.py --json-output tmp/evals.json
```

Each case runs the component once and applies the configured judge and reliability checks to the same output. The CLI returns a nonzero exit code when a case fails, so the suite can gate CI.

## Evaluation types

| Type           | Measures                                             | Guide                                                  |
| -------------- | ---------------------------------------------------- | ------------------------------------------------------ |
| Accuracy       | Correctness against an expected answer               | [Accuracy evals](/evals/accuracy/overview)             |
| Agent as judge | Custom quality criteria scored by an evaluator model | [Agent-as-judge evals](/evals/agent-as-judge/overview) |
| Reliability    | Expected tool calls and arguments                    | [Reliability evals](/evals/reliability/overview)       |
| Performance    | Runtime latency and memory use                       | [Performance evals](/evals/performance/overview)       |

## Where evals run

| Stage             | Pattern                                                                                                               |
| ----------------- | --------------------------------------------------------------------------------------------------------------------- |
| Local development | Run one case while changing an agent.                                                                                 |
| CI                | Run tagged [eval suites](/evals/suite/overview) and keep the JSON report.                                             |
| Production        | Evaluate selected outputs with a synchronous or [background post-hook](/agent-os/usage/background-output-evaluation). |
| AgentOS           | Store eval results in a configured database and manage them through the AgentOS API.                                  |

## Next steps

| Task                           | Guide                                         |
| ------------------------------ | --------------------------------------------- |
| Build an eval suite            | [Eval suites](/evals/suite/overview)          |
| Add evals to an agent platform | [Agent platform evals](/agent-platform/evals) |
| Inspect the API surface        | [Agent API](/features/api)                    |