The WebsiteKnowledgeBase reads websites, converts them into vector embeddings and loads them to a vector_db.

​
Usage

We are using a local PgVector database for this example. Make sure it’s running
pip install bs4
knowledge_base.py
from agno.knowledge.website import WebsiteKnowledgeBase
from agno.vectordb.pgvector import PgVector

knowledge_base = WebsiteKnowledgeBase(
    urls=["https://docs.agno.com/introduction"],
    # Number of links to follow from the seed URLs
    max_links=10,
    # Table name: ai.website_documents
    vector_db=PgVector(
        table_name="website_documents",
        db_url="postgresql+psycopg://ai:ai@localhost:5532/ai",
    ),
)
Then use the knowledge_base with an Agent:
agent.py
from agno.agent import Agent
from knowledge_base import knowledge_base

agent = Agent(
    knowledge=knowledge_base,
    search_knowledge=True,
)
agent.knowledge.load(recreate=False)

agent.print_response("Ask me about something from the knowledge base")

​
WebsiteKnowledgeBase also supports async loading.

pip install qdrant-client
We are using a local Qdrant database for this example. Make sure it’s running
async_knowledge_base.py
import asyncio

import asyncio

from agno.agent import Agent
from agno.knowledge.website import WebsiteKnowledgeBase
from agno.vectordb.qdrant import Qdrant

COLLECTION_NAME = "website-content"

vector_db = Qdrant(collection=COLLECTION_NAME, url="http://localhost:6333")


# Create a knowledge base with the seed URLs
knowledge_base = WebsiteKnowledgeBase(
    urls=["https://docs.agno.com/introduction"],
    # Number of links to follow from the seed URLs
    max_links=5,
    # Table name: ai.website_documents
    vector_db=vector_db,
)

# Create an agent with the knowledge base
agent = Agent(knowledge=knowledge_base, search_knowledge=True, debug_mode=True)

if __name__ == "__main__":
    # Comment out after first run
    asyncio.run(knowledge_base.aload(recreate=False))

    # Create and use the agent
    asyncio.run(agent.aprint_response("How does agno work?", markdown=True))

​
Params

ParameterTypeDefaultDescription
urlsList[str][]URLs to read
readerOptional[WebsiteReader]NoneA WebsiteReader that reads the urls and converts them into Documents for the vector database.
max_depthint3Maximum depth to crawl.
max_linksint10Number of links to crawl.
WebsiteKnowledgeBase is a subclass of the AgentKnowledge class and has access to the same params.

​
Developer Resources