AccuracyEval runs an agent or team against an input, then scores the response against an expected output with an evaluator agent on a 1-10 scale.
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
input | Union[str, Callable] | Required | Input sent to the agent or team. A callable must return the input string. |
expected_output | Union[str, Callable] | Required | Expected answer the evaluator scores against. A callable must return the expected string. |
agent | Optional[Agent] | None | Agent to evaluate. Provide exactly one of agent or team. |
team | Optional[Team] | None | Team to evaluate. Provide exactly one of agent or team. |
name | Optional[str] | None | Evaluation name. |
eval_id | str | Autogenerated UUID | Evaluation ID. |
num_iterations | int | 1 | Number of times the input is run and scored. |
model | Optional[Model] | None | Model for the default evaluator agent. Falls back to OpenAI o4-mini when unset. |
evaluator_agent | Optional[Agent] | None | Custom agent that scores the response. Replaces the default evaluator. |
additional_guidelines | Optional[Union[str, List[str]]] | None | Guidelines appended to the default evaluator’s instructions. |
additional_context | Optional[str] | None | Context appended to the default evaluator’s instructions. |
print_summary | bool | False | Print the summary table with score statistics after the run. |
print_results | bool | False | Print the per-iteration results table after the run. |
show_spinner | bool | True | Render the progress spinner. Set to False when the eval runs inside code that must not write to the console. |
file_path_to_save_results | Optional[str] | None | File path where the result is saved. |
debug_mode | bool | False | Enable debug logs. Defaults to True when the AGNO_DEBUG environment variable is "true". |
db | Optional[Union[BaseDb, AsyncBaseDb]] | None | Database where the eval run is stored. |
telemetry | bool | True | Log minimal telemetry for the eval run. |
result | Optional[AccuracyResult] | None | Result of the evaluation. Set after a run completes. |
Methods
| Method | Return type | Description |
|---|---|---|
run(*, print_summary=True, print_results=True) | Optional[AccuracyResult] | Run the agent or team num_iterations times and score each response. Raises with an async database; use arun() instead. |
arun(*, print_summary=True, print_results=True) | Optional[AccuracyResult] | Async version of run(). |
run_with_output(*, output, print_summary=True, print_results=True) | Optional[AccuracyResult] | Score the given output string instead of running the agent or team. |
arun_with_output(*, output, print_summary=True, print_results=True) | Optional[AccuracyResult] | Async version of run_with_output(). |
AccuracyResult
| Field | Type | Description |
|---|---|---|
results | List[AccuracyEvaluation] | One entry per scored iteration. |
avg_score | Optional[float] | Average score across iterations. None when there are no results. |
mean_score | Optional[float] | Mean score across iterations. Same value as avg_score. |
min_score | Optional[float] | Lowest score across iterations. |
max_score | Optional[float] | Highest score across iterations. |
std_dev_score | Optional[float] | Standard deviation of scores. 0 with a single result. |
AccuracyResult also exposes print_summary(console=None) and print_results(console=None) to render the statistics and per-iteration tables.
AccuracyEvaluation
Each entry inAccuracyResult.results is an AccuracyEvaluation:
| Field | Type | Description |
|---|---|---|
input | str | Input sent to the agent or team. |
output | str | Response produced by the agent or team. |
expected_output | str | Expected answer the response was scored against. |
score | int | Accuracy score from 1 to 10. |
reason | str | The evaluator’s reasoning for the score. |