Execute Evaluation

Database selection

Use an evaluation-capable database registered with AgentOS or its components. The example API server registers SQLite through its agent. When several database IDs are available, supply db_id; selecting table also requires db_id. These values select registered storage, not an arbitrary database connection. Authentication and configured user scoping apply to the stored evaluation records.

Evaluation inputs

Choose one local registered Agno Agent or Team with its model, credentials and tools configured. The current endpoint does not support executing evaluations against RemoteAgent or RemoteTeam. Evaluations run the selected component and can invoke its actual models and tools.

eval_typeAdditional required input
accuracyNonempty expected_output
agent_as_judgeNonempty criteria
performanceNo additional conditional field
reliabilityexpected_tool_calls; an empty list is valid

Missing conditional input returns 400. For example, this request evaluates the tool-free example agent:

curl --fail-with-body http://127.0.0.1:7777/eval-runs \
  -H "Authorization: Bearer $OS_SECURITY_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"agent_id":"reference-agent","eval_type":"reliability","input":"Say hello","expected_tool_calls":[]}'

num_iterations controls accuracy and performance evaluations. The current judge and reliability handlers each run once. warmup_runs applies to performance.

A model override requires both model_id and model_provider and a model already available to the runtime. It changes the tested component's model. The evaluator uses the component's original model or its own default, so model_id is not a separate judge-model selector.

POST/eval-runs

Run evaluation tests on agents or teams. Supports accuracy, agent-as-judge, performance, and reliability evaluations. Requires either agent_id or team_id, but not both.

Authorization

HTTPBearer
AuthorizationBearer <token>

In: header

Query Parameters

db_id?|

Database ID to use for evaluation

table?|

Table to use for evaluation

Request Body

application/json

TypeScript Definitions

Use the request body type in TypeScript.

Response Body

application/json

application/json

application/json

application/json

application/json

application/json

curl --request POST 'https://example.com/eval-runs' \  --header 'Content-Type: application/json' \  --data-raw '{"eval_type":"accuracy","input":"string"}'
{  "id": "f2b2d72f-e9e2-4f0e-8810-0a7e1ff58614",  "agent_id": "basic-agent",  "model_id": "gpt-4o",  "model_provider": "OpenAI",  "eval_type": "reliability",  "eval_data": {    "eval_status": "PASSED",    "failed_tool_calls": [],    "passed_tool_calls": [      "multiply"    ]  },  "created_at": "2025-08-27T15:41:59Z",  "updated_at": "2025-08-27T15:41:59Z"}