Skip to main content
The Web Search Reader searches and reads web search results, converting them into vector embeddings for your knowledge base.

​
Code

examples/concepts/knowledge/readers/web_search_reader_async.py
import asyncio

from agno.agent import Agent
from agno.db.postgres.postgres import PostgresDb
from agno.knowledge.knowledge import Knowledge
from agno.knowledge.reader.web_search_reader import WebSearchReader
from agno.vectordb.pgvector import PgVector

db_url = "postgresql+psycopg://ai:ai@localhost:5532/ai"

db = PostgresDb(id="web-search-db", db_url=db_url)

vector_db = PgVector(
    db_url=db_url,
    table_name="web_search_documents",
)
knowledge = Knowledge(
    name="Web Search Documents",
    contents_db=db,
    vector_db=vector_db,
)


# Initialize the Agent with the knowledge
agent = Agent(
    knowledge=knowledge,
    search_knowledge=True,
)


if __name__ == "__main__":
    # Comment out after first run
    asyncio.run(
        knowledge.add_content_async(
            topics=["web3 latest trends 2025"],
            reader=WebSearchReader(
                max_results=3,
                search_engine="duckduckgo",
                chunk=True,
            ),
        )
    )

    # Create and use the agent
    asyncio.run(
        agent.aprint_response(
            "What are the latest AI trends according to the search results?",
            markdown=True,
        )
    )

​
Usage

1

Create a virtual environment

Open the Terminal and create a python virtual environment.
python3 -m venv .venv
source .venv/bin/activate
2

Install libraries

pip install -U requests beautifulsoup4 agno openai
3

Run PgVector

docker run -d \
  -e POSTGRES_DB=ai \
  -e POSTGRES_USER=ai \
  -e POSTGRES_PASSWORD=ai \
  -e PGDATA=/var/lib/postgresql/data/pgdata \
  -v pgvolume:/var/lib/postgresql/data \
  -p 5532:5432 \
  --name pgvector \
  agno/pgvector:16
4

Set environment variables

export OPENAI_API_KEY=xxx
5

Run Agent

python examples/concepts/knowledge/readers/web_search_reader_async.py

​
Params

ParameterTypeDefaultDescription
search_timeoutint10Timeout for search operations in seconds
request_timeoutint30Timeout for HTTP requests in seconds
delay_between_requestsfloat2.0Delay between requests in seconds
max_retriesint3Maximum number of retries for failed requests
user_agentstr"Mozilla/5.0..."User agent string for HTTP requests
search_engineLiteral["duckduckgo", "google"]"duckduckgo"Search engine to use
search_delayfloat3.0Delay between search requests in seconds
max_search_retriesint2Maximum retries for search operations
rate_limit_delayfloat5.0Delay when rate limited in seconds
exponential_backoffboolTrueWhether to use exponential backoff for retries
chunking_strategyOptional[ChunkingStrategy]SemanticChunking()Strategy for chunking content