Confident AI

Confident AI is an LLM observability and evaluation platform. confident-trace, its OpenTelemetry-native tracing SDK, detects Agno automatically. Agent runs show up in the Observatory with their model and tool calls nested beneath them. No instrumentor or tracer provider configuration is required.

Setup

uv pip install -U agno openai confident-trace

export CONFIDENT_API_KEY="your_confident_api_key_here"
export OPENAI_API_KEY="your_openai_api_key_here"

from confident_trace import init, shutdown
from agno.agent import Agent
from agno.models.openai import OpenAIResponses

init()

agent = Agent(
    name="assistant",
    model=OpenAIResponses(id="gpt-5.6-luna"),
)

try:
    result = agent.run("Explain OpenTelemetry in one sentence.")
    print(result.content)
finally:
    shutdown()

Captured data

confident-trace attaches an Agno integration label to your spans and records:

• Agent and team execution: Run names, timing, status, inputs, outputs, and parent-child relationships.
• Workflows and steps: Workflow execution, individual steps, and parallel, conditional, loop, and router containers.
• Tool calls: Tool names and their input and output.
• Model calls: Messages, model details, and token usage from supported provider integrations.
• Custom spans: Application spans created inside a tool stay nested under that execution.

Sync, async, and streamed runs are supported. Consume streams fully, or close them when stopping early.

Online evals

Configure what happens to incoming traces on the Confident AI workflows page.

Workflows support:

• Evaluation rules: Evaluate incoming traces against a metric collection.
• Classifiers: Label traces by issue, sentiment, or any dimension you define so you can group or filter them later.
• Queue ingestion: Add production traces to annotation queues for manual review.
• Dataset ingestion: Turn production traces into dataset test cases you can iterate on over time.

You can create separate workflows for traces, spans, and threads.

To evaluate a component or trace manually, pass a metric collection through update_trace. See online evaluations.

Trace properties

Use trace_context to attach tags, metadata, and a user ID that you know before the run starts. It creates no extra span. The trace started by agent.run() inherits everything you pass:

from confident_trace import init, trace_context
from agno.agent import Agent
from agno.models.openai import OpenAIResponses

init()

agent = Agent(name="assistant", model=OpenAIResponses(id="gpt-5.6-luna"))

with trace_context(
    tags=["support"],
    metadata={"release": "2026-09"},
    user_id="user-42",
):
    result = agent.run("Explain OpenTelemetry in one sentence.")
    print(result.content)

Group traces into threads

Use turn() to group sequential Agno calls into one turn. Reuse the same thread ID on later turns to group them into one thread you can view and evaluate on Confident AI:

from confident_trace import init, turn
from agno.agent import Agent
from agno.models.openai import OpenAIResponses

init()

agent = Agent(name="assistant", model=OpenAIResponses(id="gpt-5.6-luna"))

with turn("support-turn", thread_id="chat-42"):
    context = agent.run("Find the relevant account details.")
    answer = agent.run(f"Summarize these details: {context.content}")
    print(answer.content)

Notes

• Initialize once, shut down once: In a long-running server, call init() at startup and shutdown() during graceful shutdown, after active agent runs finish. Do not call them per request.
• Missing traces: A missing trace usually means the process exited before spans were exported. Call shutdown() before exit, or flush() in long-running processes.
• Model spans: Agno spans describe the structure of a run. Model calls are captured by the provider integrations that init() enables alongside Agno, so keep the relevant provider integration on. Model backends that bypass those SDKs show the agent structure without LLM spans.
• Background jobs: Background job dispatch is not traced as completed agent execution. Instrument the worker that runs the job.

Developer resources

• Confident AI tracing documentation
• Runnable Agno example
• confident-trace on PyPI
• Need help integrating? Talk to a human.