Documentation Index
Fetch the complete documentation index at: https://docs.agno.com/llms.txt
Use this file to discover all available pages before exploring further.
The LLMs.txt Reader reads an llms.txt file, follows the linked documentation pages, and turns them into documents for your knowledge base.
Code
examples/basics/knowledge/concepts/readers/overview/llms_txt_reader.py
from agno.agent import Agent
from agno.knowledge.knowledge import Knowledge
from agno.knowledge.reader.llms_txt_reader import LLMsTxtReader
from agno.vectordb.pgvector import PgVector
db_url = "postgresql+psycopg://ai:ai@localhost:5532/ai"
knowledge = Knowledge(
name="LLMs.txt Docs",
vector_db=PgVector(table_name="llms_txt_docs", db_url=db_url),
)
knowledge.insert(
url="https://docs.agno.com/llms.txt",
reader=LLMsTxtReader(max_urls=10),
)
agent = Agent(
knowledge=knowledge,
search_knowledge=True,
)
agent.print_response("What is Agno?", markdown=True)
Usage
Set up your virtual environment
uv venv --python 3.12
source .venv/bin/activate
Install dependencies
uv pip install -U beautifulsoup4 sqlalchemy psycopg pgvector agno openai
Set environment variables
export OPENAI_API_KEY=xxx
Run PgVector
docker run -d \
-e POSTGRES_DB=ai \
-e POSTGRES_USER=ai \
-e POSTGRES_PASSWORD=ai \
-e PGDATA=/var/lib/postgresql/data/pgdata \
-v pgvolume:/var/lib/postgresql/data \
-p 5532:5432 \
--name pgvector \
agno/pgvector:16
Run Agent
python examples/basics/knowledge/concepts/readers/overview/llms_txt_reader.py
Params
| Parameter | Type | Default | Description |
|---|
url | str | Required | URL of the llms.txt file to read |
max_urls | int | 20 | Maximum number of linked URLs to fetch from the file |
timeout | int | 60 | HTTP timeout in seconds |
proxy | Optional[str] | None | Optional HTTP proxy URL |
skip_optional | bool | False | Skip entries under the ## Optional section |
chunking_strategy | Optional[ChunkingStrategy] | FixedSizeChunking() | Strategy for chunking content |
allowed_hosts | Optional[List[str]] | None | Hostnames the reader is allowed to fetch from. See Restricting URL Fetches. |
