> ## Documentation Index
> Fetch the complete documentation index at: https://docs.agno.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Website Reader

> Crawl websites and convert pages into knowledge base documents with WebsiteReader.

`WebsiteReader` crawls a starting page and discovered links, then creates documents from the extracted page text.

```python website_reader.py theme={null}
from agno.knowledge.reader.website_reader import WebsiteReader

reader = WebsiteReader(
    max_depth=2,
    max_links=5,
    allowed_hosts=["docs.agno.com"],
)

documents = reader.read("https://docs.agno.com/")
for document in documents:
    print(document.meta_data["url"], len(document.content))
```

## Run the Reader

<Steps>
  <Snippet file="create-venv-step.mdx" />

  <Step title="Install dependencies">
    ```bash theme={null}
    uv pip install -U beautifulsoup4 agno
    ```
  </Step>

  <Step title="Run the script">
    ```bash theme={null}
    python website_reader.py
    ```
  </Step>
</Steps>

## Reader Parameters

<Snippet file="website-reader-reference.mdx" />

<Warning>
  Set `allowed_hosts` for every crawl in v2.7.2. The fallback primary-domain check uses a hostname suffix and can admit unrelated hosts whose names end with the same text.
</Warning>

`WebsiteReader.async_read()` performs the crawl with asynchronous HTTP requests.

## Next Steps

| Task                       | Guide                                                                                   |
| -------------------------- | --------------------------------------------------------------------------------------- |
| Restrict outbound requests | [Restricting URL Fetches](/knowledge/concepts/readers/overview#restricting-url-fetches) |
| Configure content chunking | [Chunking Overview](/knowledge/concepts/chunking/overview)                              |
| Inspect the complete API   | [Website Reader Reference](/reference/knowledge/reader/website)                         |