PerformanceEval measures the run time and peak memory usage of a function, with warm-up runs excluded from the final statistics.
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
func | Callable | Required | Function to measure. Must be async when using arun(). |
measure_runtime | bool | True | Measure execution time per iteration. |
measure_memory | bool | True | Measure peak memory usage per iteration with tracemalloc, adjusted against a computed baseline. |
name | Optional[str] | None | Evaluation name. |
eval_id | str | Autogenerated UUID | Evaluation ID. |
warmup_runs | Optional[int] | 10 | Number of unmeasured runs before measurement starts. |
num_iterations | int | 50 | Number of measured iterations. Runtime and memory are measured in separate passes of this many iterations each. |
print_summary | bool | False | Print the statistics summary table after the run. |
print_results | bool | False | Print the per-iteration results table after the run. |
show_spinner | bool | True | Render the progress spinner. Set to False when the eval runs inside code that must not write to the console. |
memory_growth_tracking | bool | False | Compare memory snapshots between iterations and log growth sources. Requires debug_mode for the log output. |
top_n_memory_allocations | int | 5 | Number of memory allocation sources logged when tracking growth. |
agent_id | Optional[str] | None | Agent ID attached to the stored eval run. |
team_id | Optional[str] | None | Team ID attached to the stored eval run. |
model_id | Optional[str] | None | Model ID attached to the stored eval run. |
model_provider | Optional[str] | None | Model provider attached to the stored eval run. |
file_path_to_save_results | Optional[str] | None | File path where the result is saved. |
debug_mode | bool | False | Enable debug logs, including raw and adjusted memory readings. Defaults to True when the AGNO_DEBUG environment variable is "true". |
db | Optional[Union[BaseDb, AsyncBaseDb]] | None | Database where the eval run is stored. |
telemetry | bool | True | Log minimal telemetry for the eval run. |
result | Optional[PerformanceResult] | None | Result of the evaluation. Set after a run completes. |
Methods
| Method | Return type | Description |
|---|---|---|
run(*, print_summary=False, print_results=False, memory_growth_tracking=False) | PerformanceResult | Run warm-ups, then measure runtime and memory over num_iterations. Raises with an async database; use arun() instead. |
arun(*, print_summary=False, print_results=False, memory_growth_tracking=False) | PerformanceResult | Async version of run(). Raises when func is not a coroutine function. |
PerformanceResult
Runtime statistics are in seconds; memory statistics are in MiB. All statistics are0 when the corresponding measurement is disabled.
| Field | Type | Description |
|---|---|---|
run_times | List[float] | Measured run time per iteration. |
avg_run_time | float | Average run time. |
min_run_time | float | Fastest run time. |
max_run_time | float | Slowest run time. |
std_dev_run_time | float | Standard deviation of run times. |
median_run_time | float | Median run time. |
p95_run_time | float | 95th percentile run time. |
memory_usages | List[float] | Baseline-adjusted peak memory usage per iteration. |
avg_memory_usage | float | Average memory usage. |
min_memory_usage | float | Lowest memory usage. |
max_memory_usage | float | Highest memory usage. |
std_dev_memory_usage | float | Standard deviation of memory usage. |
median_memory_usage | float | Median memory usage. |
p95_memory_usage | float | 95th percentile memory usage. |
PerformanceResult also exposes print_summary() and print_results(), both accepting console, measure_memory, and measure_runtime arguments.