> ## Documentation Index
> Fetch the complete documentation index at: https://docs.agno.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Website Ingestion: One Row Per Page

> Load a website into a knowledge base page by page from its sitemap: one content row per page with its source URL kept, so the agent can cite the page it answered from and a re-run refreshes only pages that changed.

```python website_per_page.py theme={null}
"""
Website Ingestion: One Row Per Page
===================================
Load a website into a knowledge base page by page from its sitemap:
one content row per page with its source URL kept, so the agent can
cite the page it answered from and a re-run refreshes only pages
that changed.

SitemapReader discovers pages from the site's sitemap (robots.txt and
sitemap indexes are followed) and fetches each page whole. Pages land
as separate rows on the Knowledge page; deleting the site row removes
every page and its vectors.

A bare sitemap URL selects this reader automatically:
    await knowledge.ainsert(url="https://docs.agno.com/sitemap.xml")
"""

import asyncio

from agno.agent import Agent
from agno.db.sqlite import SqliteDb
from agno.knowledge.embedder.openai import OpenAIEmbedder
from agno.knowledge.knowledge import Knowledge
from agno.knowledge.reader.sitemap_reader import SitemapReader
from agno.models.openai import OpenAIResponses
from agno.vectordb.qdrant import Qdrant

# ---------------------------------------------------------------------------
# Create Knowledge
# ---------------------------------------------------------------------------
# The contents db is what holds the per-page rows (and the digests that make
# re-ingest refresh only changed pages) — without it, only vectors are stored.
knowledge = Knowledge(
    name="Agno Docs",
    contents_db=SqliteDb(db_file="tmp/agno_docs_contents.db"),
    vector_db=Qdrant(
        collection="website-pages",
        url="http://localhost:6333",
        embedder=OpenAIEmbedder(id="text-embedding-3-small"),
    ),
)

# ---------------------------------------------------------------------------
# Create Agent
# ---------------------------------------------------------------------------
agent = Agent(
    model=OpenAIResponses(id="gpt-5.6-luna"),
    knowledge=knowledge,
    search_knowledge=True,
    instructions="Answer from the knowledge base and cite the source URL of the page you used.",
    markdown=True,
)


# ---------------------------------------------------------------------------
# Run Demo
# ---------------------------------------------------------------------------
async def main() -> None:
    # One call: sitemap discovery, page-by-page fetch, one row per page.
    # Uses Parallel's extraction when PARALLEL_API_KEY or the mcp extra is
    # available, the built-in fetcher otherwise.
    await knowledge.ainsert(
        url="https://docs.agno.com",
        reader=SitemapReader(max_pages=25),
    )

    await agent.aprint_response("What is an Agent in Agno? Cite the page you used.")


if __name__ == "__main__":
    asyncio.run(main())
```

## Run the Example

<Steps>
  <Snippet file="create-venv-step.mdx" />

  <Step title="Install dependencies">
    ```bash theme={null}
    uv pip install -U agno beautifulsoup4 openai qdrant-client sqlalchemy
    ```
  </Step>

  <Step title="Export your OpenAI API key">
    <CodeGroup>
      ```bash Mac/Linux theme={null}
      export OPENAI_API_KEY="your_openai_api_key_here"
      ```

      ```bash Windows theme={null}
      $Env:OPENAI_API_KEY="your_openai_api_key_here"
      ```
    </CodeGroup>
  </Step>

  <Step title="Run Qdrant">
    ```bash theme={null}
    docker run -d --name qdrant -p 6333:6333 qdrant/qdrant:latest
    ```
  </Step>

  <Step title="Run the example">
    Save the code above as `website_per_page.py`, then run:

    ```bash theme={null}
    python website_per_page.py
    ```
  </Step>
</Steps>

Full source: [cookbook/07\_knowledge/01\_getting\_started/05\_website\_per\_page.py](https://github.com/agno-agi/agno/blob/v3.0.4/cookbook/07_knowledge/01_getting_started/05_website_per_page.py)