ReliabilityEval
ReliabilityEval parameters, tool call and argument checks, and ReliabilityResult fields.
ReliabilityEval checks the tool calls in a completed agent or team run against expected tool names and arguments.
from agno.eval import ReliabilityEval, ReliabilityResultParameters
| Parameter | Type | Default | Description |
|---|---|---|---|
name | Optional[str] | None | Evaluation name. |
agent_response | Optional[RunOutput] | None | Agent run to check. Provide exactly one of agent_response or team_response. |
team_response | Optional[TeamRunOutput] | None | Team run to check, including member responses. Provide exactly one of agent_response or team_response. |
expected_tool_calls | Optional[List[str]] | None | Tool names that must have clean, non-paused executions in the run. Missing names fail the evaluation. |
allow_additional_tool_calls | bool | False | Treat tool calls outside expected_tool_calls as allowed extras instead of failures. |
expected_tool_call_arguments | Optional[Dict[str, Union[Dict[str, Any], List[Dict[str, Any]]]]] | None | Expected arguments per tool name, matched as a subset of a clean execution’s actual arguments. A list of dicts requires each spec to match at least one call to that tool. |
print_results | bool | False | Print the results table after the run. |
show_spinner | bool | True | Render the progress spinner. Set to False when the eval runs inside code that must not write to the console. |
file_path_to_save_results | Optional[str] | None | File path where the result is saved. |
debug_mode | bool | False | Enable debug logs. Defaults to True when the AGNO_DEBUG environment variable is "true". |
db | Optional[Union[BaseDb, AsyncBaseDb]] | None | Database where the eval run is stored. |
telemetry | bool | True | Log minimal telemetry for the eval run. |
result | Optional[ReliabilityResult] | None | Result of the evaluation. Set after a run completes. |
Each execution creates a fresh run_id. The evaluation object’s result holds its latest returned result.
Methods
| Method | Return type | Description |
|---|---|---|
run(*, print_results=False) | Optional[ReliabilityResult] | Check the response's tool calls against the expectations. Raises with an async database; use arun() instead. |
arun(*, print_results=False) | Optional[ReliabilityResult] | Async version of run(). |
ReliabilityResult
The evaluation passes when there are no failed tool calls, no missing tool calls, and no failed argument checks.
| Field | Type | Description |
|---|---|---|
run_id | str | Unique ID generated for this evaluation execution. |
eval_status | str | "PASSED" or "FAILED". |
failed_tool_calls | List[str] | Tool calls outside expected_tool_calls when additional calls are not allowed. |
passed_tool_calls | List[str] | Clean, non-paused executions that matched expected_tool_calls. |
additional_tool_calls | List[str] | Tool calls outside expected_tool_calls, recorded when allow_additional_tool_calls=True. |
missing_tool_calls | List[str] | Expected names without a clean, non-paused execution, including requested-only, refused, or errored calls. |
failed_argument_checks | List[str] | Tool names whose arguments did not match expected_tool_call_arguments. |
passed_argument_checks | List[str] | Tool names with clean executions whose arguments matched expected_tool_call_arguments. |
ReliabilityResult also exposes print_eval(console=None) to render the results table, and assert_passed(), which asserts eval_status == "PASSED" for use in test suites.