Website Reader

WebsiteReader crawls a starting page and discovered links, then creates documents from the extracted page text.

from agno.knowledge.reader.website_reader import WebsiteReader

reader = WebsiteReader(
    max_depth=2,
    max_links=5,
    allowed_hosts=["docs.agno.com"],
)

documents = reader.read("https://docs.agno.com/")
for document in documents:
    print(document.meta_data["url"], len(document.content))

Run the Reader

Set up your virtual environment

uv venv --python 3.12
source .venv/bin/activate

Install dependencies

uv pip install -U beautifulsoup4 agno

Run the script

python website_reader.py

Reader Parameters

Parameter	Type	Default	Description
`chunking_strategy`	`Optional[ChunkingStrategy]`	`FixedSizeChunking()`	Strategy used to chunk documents
`max_depth`	`int`	`3`	Maximum depth level for crawling links
`max_links`	`int`	`10`	Maximum number of links to crawl
`timeout`	`int`	`10`	Request timeout in seconds
`proxy`	`Optional[str]`	`None`	Proxy URL used for requests
`allowed_hosts`	`Optional[List[str]]`	`None`	Hostnames the reader is allowed to fetch from. See Restricting URL Fetches.

WebsiteReader also accepts the base Reader constructor parameters.

Set allowed_hosts for every crawl. The fallback primary-domain check uses a hostname suffix and can admit unrelated hosts whose names end with the same text.

WebsiteReader.async_read() performs the crawl with asynchronous HTTP requests.

Next Steps

Task	Guide
Restrict outbound requests	Restricting URL Fetches
Configure content chunking	Chunking Overview
Inspect the complete API	Website Reader Reference