Usage

Basic

Test a vegetarian recipe agent with Scenario's user simulator and judge agents in a pytest suite.

Use the Scenario framework for agentic simulation-based testing. Scenario simulates conversations between agents, user simulators, and judges so you can test and evaluate agent behavior in a controlled environment.

Tip: See the Agno integration guide for a more complex agent with tool calls and advanced scenario features.

Code

The adapter stores history in memory for this test process and includes it on later turns with the same thread ID.

scenario_testing.py
import pytest
import scenario
from agno.agent import Agent
from agno.db.in_memory import InMemoryDb
from agno.models.openai import OpenAIResponses

# Configure Scenario defaults (model for user simulator and judge)
scenario.configure(default_model="openai/gpt-4.1-mini")

@pytest.mark.agent_test
@pytest.mark.asyncio
async def test_vegetarian_recipe_agent() -> None:
    # 1. Define an AgentAdapter to wrap your agent
    class VegetarianRecipeAgentAdapter(scenario.AgentAdapter):
        agent: Agent

        def __init__(self) -> None:
            self.agent = Agent(
                model=OpenAIResponses(id="gpt-5.2"),
                db=InMemoryDb(),
                add_history_to_context=True,
                markdown=True,
                debug_mode=True,
                instructions="You are a vegetarian recipe agent.",
            )

        async def call(self, input: scenario.AgentInput) -> scenario.AgentReturnTypes:
            response = await self.agent.arun(
                input=input.last_new_user_message_str(), # Pass only the last user message
                session_id=input.thread_id, # Reuse the stored session for this simulated conversation
            )
            return response.content

    # 2. Run the scenario simulation
    result = await scenario.run(
        name="dinner recipe request",
        description="User is looking for a vegetarian dinner idea.",
        agents=[
            VegetarianRecipeAgentAdapter(),
            scenario.UserSimulatorAgent(),
            scenario.JudgeAgent(
                criteria=[
                    "Agent should not ask more than two follow-up questions",
                    "Agent should generate a recipe",
                    "Recipe should include a list of ingredients",
                    "Recipe should include step-by-step cooking instructions",
                    "Recipe should be vegetarian and not include any sort of meat",
                ]
            ),
        ],
    )

    # 3. Assert and inspect the result
    assert result.success

Usage

Set up your virtual environment

uv venv --python 3.12
source .venv/bin/activate

Set your API key

export OPENAI_API_KEY=xxx
export LANGWATCH_API_KEY=xxx # Optional, required for Simulation monitoring

Install dependencies

uv pip install -U openai agno langwatch-scenario pytest pytest-asyncio
# or
uv add agno langwatch-scenario openai pytest pytest-asyncio

Run Agent

pytest scenario_testing.py