Eval Suite - Agno

The eval suite runner executes declared Cases sequentially and returns a SuiteResult. Each case runs one input against one agent or team, then applies an optional judge check (AgentAsJudgeEval via criteria) and an optional reliability check (ReliabilityEval via expected_tool_calls).

from agno.eval import Case, CaseResult, JudgeMode, SuiteResult, run_cases, arun_cases, cli, acli

Case

A Case describes a single input and its checks. It’s a frozen dataclass; construction raises ValueError unless exactly one of agent or team is set and at least one check (criteria or expected_tool_calls) is configured.

Parameter	Type	Default	Description
`name`	`str`	Required	Case name. Used by the `--name` selector and in results.
`input`	`str`	Required	Input sent to the agent or team.
`agent`	`Optional[Agent]`	`None`	Agent under test. Provide exactly one of `agent` or `team`.
`team`	`Optional[Team]`	`None`	Team under test. Provide exactly one of `agent` or `team`.
`tags`	`Tuple[str, ...]`	`()`	Tags used by the `--tag` selector.
`timeout_seconds`	`Optional[int]`	`None`	Per-case timeout in seconds. Falls back to the runner’s `default_timeout`.
`criteria`	`Optional[str]`	`None`	Judge criteria. Setting this enables an `AgentAsJudgeEval` check on the response.
`judge_model`	`Optional[Model]`	`None`	Per-case judge model. Falls back to the runner’s `judge_model`, then the `AgentAsJudgeEval` default.
`judge_mode`	`JudgeMode`	`JudgeMode.BINARY`	Judge scoring mode. `BINARY` is a pass/fail verdict; `NUMERIC` grades 1-10 and passes when the score meets `judge_threshold`.
`judge_threshold`	`int`	`7`	Pass bar for numeric scoring, from 1 to 10. Read only when `judge_mode` is `NUMERIC`.
`expected_tool_calls`	`Optional[Tuple[str, ...]]`	`None`	Tool names that must fire during the run. Setting this enables a `ReliabilityEval` check.
`allow_additional_tool_calls`	`bool`	`True`	Allow tool calls outside `expected_tool_calls` (subset matching).
`setup`	`Optional[Callable[[], Any]]`	`None`	Runs before the case, outside the timeout. Its return value is passed to `teardown` as context. Sync callables run in a thread; async callables are awaited.
`teardown`	`Optional[Callable[[Any, CaseResult], Any]]`	`None`	Runs after the case whenever `setup` completed, including on failure, error, and timeout. Receives `(context, result)`.

JudgeMode

JudgeMode is a string enum whose values match AgentAsJudgeEval.scoring_strategy. The plain strings "binary" and "numeric" work anywhere a JudgeMode does.

Member	Value	Description
`JudgeMode.BINARY`	`"binary"`	The judge returns a pass/fail verdict.
`JudgeMode.NUMERIC`	`"numeric"`	The judge returns a 1-10 score. The case passes when the score meets `judge_threshold`.

run_cases and arun_cases

def run_cases(
    cases: Sequence[Case],
    *,
    tag: Optional[str] = None,
    name: Optional[str] = None,
    default_timeout: int = 120,
    judge_model: Optional[Model] = None,
    db: Optional[Union[BaseDb, AsyncBaseDb]] = None,
    on_case_start: Optional[Callable[[Case], None]] = None,
    on_case_end: Optional[Callable[[Case, CaseResult], None]] = None,
    on_run_event: Optional[Callable[[Case, RunOutputEvent], None]] = None,
) -> SuiteResult

arun_cases takes the same arguments and is awaited. run_cases is a sync wrapper that runs the whole suite on a single event loop via asyncio.run.

Argument	Type	Default	Description
`cases`	`Sequence[Case]`	Required	Cases to select from.
`tag`	`Optional[str]`	`None`	Keep only cases with this tag.
`name`	`Optional[str]`	`None`	Keep only the case with this name.
`default_timeout`	`int`	`120`	Per-case timeout in seconds when `Case.timeout_seconds` is `None`.
`judge_model`	`Optional[Model]`	`None`	Suite-wide default judge model. `Case.judge_model` overrides it.
`db`	`Optional[Union[BaseDb, AsyncBaseDb]]`	`None`	Passed to the judge and reliability evals so results log to storage.
`on_case_start`	`Optional[Callable[[Case], None]]`	`None`	Presentation hook called before each case runs.
`on_case_end`	`Optional[Callable[[Case, CaseResult], None]]`	`None`	Presentation hook called with each case and its `CaseResult`, including skipped cases after an abort.
`on_run_event`	`Optional[Callable[[Case, RunOutputEvent], None]]`	`None`	Presentation hook called with every streamed run event.

The runner performs no console I/O; all presentation flows through the hooks. Hooks must be sync callables invoked on the event loop. A hook that raises, or an async hook, is recorded as a hook: error on the case without aborting the suite. A cancelled run aborts the suite; the unrun cases are appended as failed with skipped=True so the payload accounts for every selected case.

CaseResult

Field	Type	Description
`name`	`str`	Case name.
`tags`	`Tuple[str, ...]`	Case tags.
`agent_id`	`Optional[str]`	Agent ID for an agent case. `None` for a team case.
`team_id`	`Optional[str]`	Team ID for a team case. `None` for an agent case.
`session_id`	`str`	Generated eval session ID linking the case to its stored session and trace. Empty for skipped cases.
`duration_seconds`	`float`	Wall-clock duration of the case, including setup and teardown.
`judge_passed`	`Optional[bool]`	Judge verdict. `None` when no `criteria` was configured.
`judge_reason`	`Optional[str]`	The judge’s stated reason, when available.
`judge_score`	`Optional[int]`	Numeric-mode score from 1 to 10. `None` in binary mode.
`reliability_passed`	`Optional[bool]`	Reliability verdict. `None` when no `expected_tool_calls` was configured.
`output`	`Optional[str]`	Response text the judge graded.
`tools_called`	`Tuple[str, ...]`	Tool names fired during the run, in order, including team member tools.
`timed_out`	`bool`	Whether the case exceeded its timeout.
`skipped`	`bool`	`True` for cases the suite never ran after a cancelled-run abort.
`error`	`Optional[str]`	Run, judge, reliability, hook, and teardown errors, joined with `"; "`.
`response`	`Optional[Union[RunOutput, TeamRunOutput]]`	Raw run output for programmatic access. Excluded from `to_dict()`.
`passed`	`bool` (property)	`True` when there is no error and every configured check passed. `False` when no check ran.

SuiteResult

Member	Type	Description
`results`	`List[CaseResult]`	One entry per selected case, including skipped cases.
`total`	`int` (property)	Number of case results.
`passed`	`int` (property)	Number of passing cases.
`failed`	`int` (property)	Number of failing cases.
`status`	`str` (property)	`"PASS"` when all cases passed. An empty suite is `"FAIL"`, so a typo’d selector cannot green-light CI.
`to_dict()`	`Dict[str, Any]`	Machine-readable payload with a `summary` block and per-case entries. This shape is a stable contract for CI consumers.

cli and acli

def cli(
    cases: Sequence[Case],
    *,
    db: Optional[Union[BaseDb, AsyncBaseDb]] = None,
    judge_model: Optional[Model] = None,
    default_timeout: int = 120,
    argv: Optional[Sequence[str]] = None,
) -> int

cli runs an argparse CLI over the given cases and returns the exit code. Call it from a script’s __main__ block with sys.exit(cli(CASES)). acli takes the same arguments and is the awaitable variant for callers already inside an event loop. When argv is None, arguments are read from sys.argv.

Flags

Flag	Description
`--name NAME`	Run only the case with this name.
`--tag TAG`	Run only cases with this tag.
`--timeout SECONDS`	Default per-case timeout in seconds. Defaults to the `default_timeout` argument.
`--json-output PATH`	Write the machine-readable JSON results to this path.
`--list`	List the selected cases without running them. With `--json-output`, writes the case list instead of results.
`-v`, `--verbose`	Render the full run panels (Message, Tool Calls, Response) after each case.

Exit codes

Code	Meaning
`0`	All selected cases passed.
`1`	Any case failed, or the `--json-output` write failed.
`2`	No cases matched the selector, or an argparse usage error.

​Case

​JudgeMode

​run_cases and arun_cases

​CaseResult

​SuiteResult

​cli and acli

​Flags

​Exit codes