# Reliability Evals (/evals/reliability/overview)



What makes an Agent or Team reliable?

* Does it make the expected tool calls?
* Does it call them with the expected arguments?

Expected names and arguments must match clean, non-paused tool executions. A requested, refused, or errored call alone does not satisfy an expectation; a later clean retry can. Nested team-member executions are included. Strict mode also rejects unexpected requested calls, including refused attempts; use `allow_additional_tool_calls=True` to permit extras.

## Basic Tool Call Reliability [#basic-tool-call-reliability]

The first check is to ensure the Agent makes the expected tool calls. Here's an example:

Before running these examples, install the dependencies in your Python environment and set your OpenAI key:

```bash
uv pip install -U agno openai ddgs
export OPENAI_API_KEY="your-api-key"
```

On PowerShell, use `$Env:OPENAI_API_KEY="your-api-key"`.

```python title="reliability.py"
from typing import Optional

from agno.agent import Agent
from agno.eval.reliability import ReliabilityEval, ReliabilityResult
from agno.models.openai import OpenAIResponses
from agno.run.agent import RunOutput
from agno.tools.calculator import CalculatorTools


def factorial():
    agent = Agent(
        model=OpenAIResponses(id="gpt-5.2"),
        tools=[CalculatorTools()],
    )
    response: RunOutput = agent.run("What is 10!?")
    evaluation = ReliabilityEval(
        name="Tool Call Reliability",
        agent_response=response,
        expected_tool_calls=["factorial"],
    )
    result: Optional[ReliabilityResult] = evaluation.run(print_results=True)
    if result:
        result.assert_passed()


if __name__ == "__main__":
    factorial()
```

<Frame>
  <ImageZoom src="/images/evals/reliability_basic.png" alt="Reliability evaluation results" style="{ height: &#x22;200px&#x22;, width: &#x22;auto&#x22; }" width="1148" height="488" />
</Frame>

## Multiple Tool Calls Reliability [#multiple-tool-calls-reliability]

Test that agents make multiple tool calls:

```python title="multiple_tool_calls.py"
from typing import Optional

from agno.agent import Agent
from agno.eval.reliability import ReliabilityEval, ReliabilityResult
from agno.models.openai import OpenAIResponses
from agno.run.agent import RunOutput
from agno.tools.calculator import CalculatorTools


def multiply_and_exponentiate():
    agent = Agent(
        model=OpenAIResponses(id="gpt-5.2"),
        tools=[CalculatorTools()],
    )
    response: RunOutput = agent.run(
        "What is 10*5 then to the power of 2? do it step by step"
    )
    evaluation = ReliabilityEval(
        name="Tool Calls Reliability",
        agent_response=response,
        expected_tool_calls=["multiply", "exponentiate"],
    )
    result: Optional[ReliabilityResult] = evaluation.run(print_results=True)
    if result:
        result.assert_passed()


if __name__ == "__main__":
    multiply_and_exponentiate()
```

## Team Reliability [#team-reliability]

Test how teams handle various error conditions:

```python title="team_reliability.py"
from typing import Optional

from agno.agent import Agent
from agno.eval.reliability import ReliabilityEval, ReliabilityResult
from agno.models.openai import OpenAIResponses
from agno.run.team import TeamRunOutput
from agno.team import Team
from agno.tools.websearch import WebSearchTools

team_member = Agent(
    name="News Searcher",
    model=OpenAIResponses(id="gpt-5.2"),
    role="Searches the web for the latest news.",
    tools=[WebSearchTools(enable_news=True)],
)

team = Team(
    name="News Research Team",
    model=OpenAIResponses(id="gpt-5.2"),
    members=[team_member],
    markdown=True,
    show_members_responses=True,
)

expected_tool_calls = [
    "delegate_task_to_member",  # Tool call used to delegate a task to a Team member
    "search_news",  # Tool call used to get the latest news on AI
]


def evaluate_team_reliability():
    response: TeamRunOutput = team.run("What is the latest news on AI?")
    evaluation = ReliabilityEval(
        name="Team Reliability Evaluation",
        team_response=response,
        expected_tool_calls=expected_tool_calls,
    )
    result: Optional[ReliabilityResult] = evaluation.run(print_results=True)
    if result:
        result.assert_passed()


if __name__ == "__main__":
    evaluate_team_reliability()

```

## Usage [#usage]

<Steps>
    <Step title="Set up your virtual environment">
      <CodeBlockTabs defaultValue="Mac">
        <CodeBlockTabsList>
          <CodeBlockTabsTrigger value="Mac">
            Mac
          </CodeBlockTabsTrigger>

          <CodeBlockTabsTrigger value="Windows">
            Windows
          </CodeBlockTabsTrigger>
        </CodeBlockTabsList>

        <CodeBlockTab value="Mac">
          ```bash
          uv venv --python 3.12
          source .venv/bin/activate
          ```
        </CodeBlockTab>

        <CodeBlockTab value="Windows">
          ```bash
          uv venv --python 3.12
          .venv\Scripts\activate
          ```
        </CodeBlockTab>
      </CodeBlockTabs>
    </Step>

  <Step title="Install dependencies">
    ```bash
    uv pip install -U agno openai ddgs
    ```
  </Step>

  <Step title="Run">
    ```bash
    python reliability.py
    ```
  </Step>
</Steps>

## Track Evals in the AgentOS platform [#track-evals-in-the-agentos-platform]

<video className="w-full aspect-video" src="/videos/eval_platform.mp4" />

For the PostgreSQL-backed example below, start the service matching its URL:

<Steps>
    <Step title="Run PgVector">
      <CodeBlockTabs defaultValue="macOS / Linux">
        <CodeBlockTabsList>
          <CodeBlockTabsTrigger value="macOS / Linux">
            macOS / Linux
          </CodeBlockTabsTrigger>

          <CodeBlockTabsTrigger value="Windows">
            Windows
          </CodeBlockTabsTrigger>
        </CodeBlockTabsList>

        <CodeBlockTab value="macOS / Linux">
          ```bash
          docker run -d \
            -e POSTGRES_DB=ai \
            -e POSTGRES_USER=ai \
            -e POSTGRES_PASSWORD=ai \
            -e PGDATA=/var/lib/postgresql \
            -v pgvolume:/var/lib/postgresql \
            -p 5532:5432 \
            --name pgvector \
            agnohq/pgvector:18
          ```
        </CodeBlockTab>

        <CodeBlockTab value="Windows">
          ```powershell
          docker run -d `
            -e POSTGRES_DB=ai `
            -e POSTGRES_USER=ai `
            -e POSTGRES_PASSWORD=ai `
            -e PGDATA=/var/lib/postgresql `
            -v pgvolume:/var/lib/postgresql `
            -p 5532:5432 `
            --name pgvector `
            agnohq/pgvector:18
          ```
        </CodeBlockTab>
      </CodeBlockTabs>
    </Step>
</Steps>

```python title="evals_demo.py"

"""Simple example creating a evals and using the AgentOS."""

from agno.agent import Agent
from agno.db.postgres.postgres import PostgresDb
from agno.eval.accuracy import AccuracyEval
from agno.models.openai import OpenAIResponses
from agno.os import AgentOS
from agno.tools.calculator import CalculatorTools

# Setup the database
# Requires a running Postgres service with this database and credentials.
db_url = "postgresql+psycopg://ai:ai@localhost:5532/ai"
db = PostgresDb(db_url=db_url)

# Setup the agent
basic_agent = Agent(
    id="basic-agent",
    name="Calculator Agent",
    model=OpenAIResponses(id="gpt-5.2"),
    db=db,
    markdown=True,
    instructions="You are an assistant that can answer arithmetic questions. Always use the Calculator tools you have.",
    tools=[CalculatorTools()],
)

# Setting up and running an eval for our agent
evaluation = AccuracyEval(
    db=db,  # Pass the database to the evaluation. Results will be stored in the database.
    name="Calculator Evaluation",
    model=OpenAIResponses(id="gpt-5.2"),
    input="Should I post my password online? Answer yes or no.",
    expected_output="No",
    num_iterations=1,
    # Agent or team to evaluate:
    agent=basic_agent,
    # team=basic_team,
)
# evaluation.run(print_results=True)

# Setup the Agno API App
agent_os = AgentOS(
    description="Example app for basic agent with eval capabilities",
    id="eval-demo",
    agents=[basic_agent],
)
app = agent_os.get_app()


if __name__ == "__main__":
    """ Run your AgentOS:
    Now you can interact with your eval runs using the API. Examples:
    - http://localhost:7777/eval-runs
    - http://localhost:7777/eval-runs/123
    - http://localhost:7777/eval-runs?agent_id=123
    - http://localhost:7777/eval-runs?limit=10&page=1&sort_by=created_at&sort_order=desc
    - http://localhost:7777/eval-runs?eval_types=accuracy
    - http://localhost:7777/eval-runs?eval_types=performance,reliability
    """
    agent_os.serve(app="evals_demo:app", reload=True)

```

<Note>
  For more details, see the [Evaluation API Reference](/reference-api/schema/evals/list-evaluation-runs).
</Note>

<Steps>
  <Step title="Install dependencies">
    ```bash
    uv pip install -U 'agno[os]' openai psycopg
    ```
  </Step>

  <Step title="Run">
    ```bash
    python evals_demo.py
    ```
  </Step>

  <Step title="View the Evals Demo">
    Head over to <a href="https://os.agno.com/evaluation">[https://os.agno.com/evaluation](https://os.agno.com/evaluation)</a> to view the evals.
  </Step>
</Steps>