Skip to main content
When an agent answers wrong, you need to know why: which tool it called, what it passed in, what the model actually saw. AgentOS captures all of it through OpenTelemetry-compatible tracing, persisted in your db. 
from agno.os import AgentOS

agent_os = AgentOS(
    agents=[agent],
    db=db,
    tracing=True,
)
Every run produces a trace tree: spans for the LLM call, each tool, hook, retrieval, and team delegation. Stored in agno_traces and agno_spans in the same db you use for sessions.

​
Own your traces

Traces are your training data. Every model call, every tool result, every retrieval, every step of every run — that’s the corpus you’d use to fine-tune, evaluate, or build your next agent on. Send it to a third-party observability platform as the only store and you’ve handed over the most valuable data your product generates. AgentOS writes traces to your database by default. You own the schema, the retention, the access patterns. You can still ship a copy to Langfuse or Arize for their dashboards (we even make that easy, see below), but your primary store is yours.

​
What gets captured

SpanAttributes
Runagent_id, user_id, session_id, model, latency, status
LLM callModel, prompt tokens, completion tokens, temperature, tool calls returned
ToolTool name, arguments, result, duration, exception (if any)
Pre/post hookHook name, duration, modified input/output
RetrievalQuery, vector store, k, returned docs, scores
Team delegationMember name, mode, sub-run trace
Traces follow OpenTelemetry semantic conventions where they exist. The data model is open, so you can query it directly: 
-- Top-10 slowest tools last hour
SELECT
    span_name,
    AVG(end_time - start_time) AS avg_duration,
    COUNT(*) AS calls
FROM agno_spans
WHERE span_name LIKE 'tool.%'
  AND start_time > NOW() - INTERVAL '1 hour'
GROUP BY span_name
ORDER BY avg_duration DESC
LIMIT 10;

​
In the AgentOS UI

The control plane renders the same traces visually. Click a run and see the full tree: LLM hops, tool calls with their inputs and outputs, hooks, sub-agent traces. Filter by user, session, time range.

​
Multi-database tracing

Traces are high-volume and write-heavy. For production deployments, you’ll often want them in a separate database from sessions and memory — different cost profile, different retention, different access patterns: 
from agno.db.postgres import PostgresDb

primary_db = PostgresDb(db_url="postgresql://primary/...")
trace_db = PostgresDb(db_url="postgresql://traces/...")

agent_os = AgentOS(
    agents=[agent],
    db=primary_db,
    tracing=True,
    trace_db=trace_db,
)
See Multi-DB tracing for the full setup.

​
External providers

Send traces to Langfuse, Langsmith, Arize, or any OpenTelemetry endpoint by adding an exporter. AgentOS still writes to your database first; the exporter is a copy, not a redirect. The full list lives in the Observability section: Langfuse, Langsmith, Arize, Langtrace, Logfire, MLflow, Maxim.

​
Run history

Beyond traces, every run is a row in agno_sessions. Pull a user’s last 100 runs, replay them, compare model versions, build dashboards: 
SELECT id, agent_id, model, status, created_at, total_tokens
FROM agno_sessions
WHERE user_id = 'user-123'
ORDER BY created_at DESC
LIMIT 100;
Same shape works for teams (agno_team_sessions) and workflows (agno_workflow_sessions).

​
Traces are not audit logs

Traces tell you what happened. Audit logs tell you what was approved. They have different retention, different access patterns, and different consumers — and conflating them gets expensive (keeping every trace forever) or risky (deleting audit data your compliance team needs).
TracesAudit logs
What they captureEvery step of every runApproved actions and who approved them
PurposeDebugging, optimization, training dataAccountability, compliance
Retention30-120 days typicalLifetime of the product
Storageagno_traces, agno_spansagno_approvals plus your own audit table
For approved actions, see Human Approval. For custom audit shapes, use post-hooks (below).

​
Audit logs via post-hooks

When you need an audit trail with custom shape (regulatory compliance, internal SOX-style logging), add a post-hook that writes to your own audit table: 
from agno.hooks import hook

@hook(run_in_background=True)
def audit_log(run_output, agent):
    write_audit_event(
        timestamp=run_output.created_at,
        agent_id=agent.id,
        user_id=run_output.user_id,
        action="agent_run",
        prompt=run_output.input,
        response=run_output.content,
        tools=[t.name for t in run_output.tool_calls],
    )

agent = Agent(..., post_hooks=[audit_log])
run_in_background=True keeps the audit write off the response path. See Hooks for the full pattern.

​
What to watch in production

SignalWhere
Run latencyagno_spans end_time - start_time per run
Token spend by modelagno_sessions total_tokens grouped by model
Tool failure rateagno_spans where status='error' and span_name LIKE 'tool.%'
User-level activityagno_sessions group by user_id, created_at::date
HITL queue depthagno_approvals where status='pending'

​
Next

Security & Auth →