Agent as Judge Evals

Agent as Judge evaluations let you define custom quality criteria and use an LLM to score your Agent’s responses. You provide evaluation criteria (like “professional tone”, “factual accuracy”, or “user-friendliness”), and an evaluator model assesses how well the Agent’s output meets those standards.

Basic Example

Here, AgentAsJudgeEval takes the Agent’s input and output and scores the response against the criteria you set.

agent_as_judge.py

from agno.agent import Agent
from agno.db.sqlite import SqliteDb
from agno.eval.agent_as_judge import AgentAsJudgeEval
from agno.models.openai import OpenAIResponses

# Setup database to persist eval results
db = SqliteDb(db_file="tmp/agent_as_judge_basic.db")

agent = Agent(
    model=OpenAIResponses(id="gpt-5.2"),
    instructions="You are a technical writer. Explain concepts clearly and concisely.",
    db=db,
)

response = agent.run("Explain what an API is")

evaluation = AgentAsJudgeEval(
    name="Explanation Quality",
    criteria="Explanation should be clear, beginner-friendly, and use simple language",
    scoring_strategy="numeric",  # Score 1-10
    threshold=7,  # Pass if score >= 7
    db=db,
)

result = evaluation.run(
    input="Explain what an API is",
    output=str(response.content),
    print_results=True,
)

Custom Evaluator Agent

You can use a custom agent to evaluate responses with specific instructions:

agent_as_judge_custom_evaluator.py

from agno.agent import Agent
from agno.eval.agent_as_judge import AgentAsJudgeEval
from agno.models.openai import OpenAIResponses

agent = Agent(
    model=OpenAIResponses(id="gpt-5.2"),
    instructions="Explain technical concepts simply.",
)

response = agent.run("Explain what an API is")

# Create a custom evaluator with specific instructions
custom_evaluator = Agent(
    model=OpenAIResponses(id="gpt-5.2"),
    description="Strict technical evaluator",
    instructions="You are a strict evaluator. Only pass exceptionally clear and accurate explanations.",
)

evaluation = AgentAsJudgeEval(
    name="Technical Accuracy",
    criteria="Explanation must be technically accurate and comprehensive",
    evaluator_agent=custom_evaluator,
)

result = evaluation.run(
    input="Explain what an API is",
    output=str(response.content),
    print_results=True,
    print_summary=True,
)

Params

Parameter	Type	Default	Description
`criteria`	`str`	`""`	The evaluation criteria describing what makes a good response (required).
`scoring_strategy`	`Literal["numeric", "binary"]`	`"binary"`	Scoring mode: `"numeric"` (1-10 scale) or `"binary"` (pass/fail).
`threshold`	`int`	`7`	Minimum score to pass (only used for numeric strategy).
`on_fail`	`Optional[Callable]`	`None`	Callback function triggered when evaluation fails.
`additional_guidelines`	`Optional[Union[str, List[str]]]`	`None`	Extra evaluation guidelines beyond the main criteria.
`name`	`Optional[str]`	`None`	Name for the evaluation.
`model`	`Optional[Model]`	`None`	Model to use for judging (defaults to gpt-5-mini if not provided).
`evaluator_agent`	`Optional[Agent]`	`None`	Custom agent to use as evaluator.
`print_summary`	`bool`	`False`	Print summary of evaluation results.
`print_results`	`bool`	`False`	Print detailed evaluation results.
`show_spinner`	`bool`	`True`	Show a progress spinner while the eval runs.
`file_path_to_save_results`	`Optional[str]`	`None`	File path to save evaluation results.
`debug_mode`	`bool`	`False`	Enable debug mode for detailed logging.
`db`	`Optional[Union[BaseDb, AsyncBaseDb]]`	`None`	Database to store evaluation results.
`telemetry`	`bool`	`True`	Enable telemetry.
`run_in_background`	`bool`	`False`	Run evaluation as background task (non-blocking).

Methods

run() / arun()

Run the evaluation synchronously (run()) or asynchronously (arun()).

Parameter	Type	Default	Description
`input`	`Optional[str]`	`None`	Input text for single evaluation.
`output`	`Optional[str]`	`None`	Output text for single evaluation.
`cases`	`Optional[List[Dict[str, str]]]`	`None`	List of input/output pairs for batch evaluation.
`print_summary`	`bool`	`False`	Print summary of evaluation results.
`print_results`	`bool`	`False`	Print detailed evaluation results.

Provide either (input, output) for single evaluation OR cases for batch evaluation, not both.

Run in a Suite

To gate many judge checks in CI, declare a Case per input with criteria and run them as a suite. judge_mode and judge_threshold on the Case map to scoring_strategy and threshold here, with the same defaults. See Eval Suites.

Agent as Judge Evals

Basic Example

Custom Evaluator Agent

Params

Methods

run() / arun()

Run in a Suite

Examples

Basic Agent as Judge

Agent as Judge as Post-Hook

Developer Resources

​Basic Example

​Custom Evaluator Agent

​Params

​Methods

​run() / arun()

​Run in a Suite

​Examples

Basic Agent as Judge

Agent as Judge as Post-Hook

​Developer Resources

Basic Example

Custom Evaluator Agent

Params

Methods

run() / arun()

Run in a Suite

Examples

Developer Resources