LLMs.txt Reader

LLMsTxtReader creates a document for an llms.txt overview, then fetches up to max_urls linked pages.

from agno.agent import Agent
from agno.knowledge.knowledge import Knowledge
from agno.knowledge.reader.llms_txt_reader import LLMsTxtReader
from agno.models.openai import OpenAIResponses
from agno.vectordb.pgvector import PgVector

db_url = "postgresql+psycopg://ai:ai@localhost:5532/ai"

knowledge = Knowledge(
    name="LLMs.txt Docs",
    vector_db=PgVector(table_name="llms_txt_docs", db_url=db_url),
)

agent = Agent(
    model=OpenAIResponses(id="gpt-5.2"),
    knowledge=knowledge,
    search_knowledge=True,
)

if __name__ == "__main__":
    knowledge.insert(
        url="https://docs.agno.com/llms.txt",
        reader=LLMsTxtReader(
            max_urls=10,
            allowed_hosts=["docs.agno.com"],
        ),
    )
    agent.print_response("What is Agno?", markdown=True)

Run the Agent

Set up your virtual environment

uv venv --python 3.12
source .venv/bin/activate

Install dependencies

uv pip install -U agno beautifulsoup4 openai pgvector psycopg sqlalchemy

Export the API key

export OPENAI_API_KEY=your_openai_api_key_here

Run PgVector

docker run -d \
  -e POSTGRES_DB=ai \
  -e POSTGRES_USER=ai \
  -e POSTGRES_PASSWORD=ai \
  -e PGDATA=/var/lib/postgresql \
  -v pgvolume:/var/lib/postgresql \
  -p 5532:5432 \
  --name pgvector \
  agnohq/pgvector:18

Run the agent

python llms_txt_reader.py

Reader Parameters

Parameter	Type	Default	Description
`max_urls`	`int`	`20`	Maximum number of linked URLs to fetch from the file
`timeout`	`int`	`60`	HTTP timeout in seconds
`proxy`	`Optional[str]`	`None`	Optional HTTP proxy URL
`skip_optional`	`bool`	`False`	Skip entries under the `## Optional` section
`chunking_strategy`	`Optional[ChunkingStrategy]`	`FixedSizeChunking()`	Strategy for chunking content
`allowed_hosts`	`Optional[List[str]]`	`None`	Hostnames the reader is allowed to fetch from. See Restricting URL Fetches.

LLMsTxtReader also accepts the base Reader constructor parameters.

allowed_hosts applies to the index, linked pages, and redirects. LLMsTxtReader.async_read() fetches the linked pages concurrently.

Next Steps

Task	Guide
Restrict outbound requests	Restricting URL Fetches
Crawl pages without an llms.txt index	Website Reader
Configure content chunking	Chunking Overview