The Web Search Reader searches and reads web search results, converting them into vector embeddings for your knowledge base.

​
Code

examples/concepts/knowledge/readers/web_search_reader.py
from agno.agent import Agent
from agno.db.postgres.postgres import PostgresDb
from agno.knowledge.embedder.openai import OpenAIEmbedder
from agno.knowledge.knowledge import Knowledge
from agno.knowledge.reader.web_search_reader import WebSearchReader
from agno.models.openai import OpenAIChat
from agno.vectordb.pgvector import PgVector

db_url = "postgresql+psycopg://ai:ai@localhost:5532/ai"
db = PostgresDb(id="web-search-db", db_url=db_url)

vector_db = PgVector(
    db_url=db_url,
    table_name="web_search_documents",
)
knowledge = Knowledge(
    name="Web Search Documents",
    contents_db=db,
    vector_db=vector_db,
)


# Load knowledge from web search
knowledge.add_content(
    topics=["agno"],
    reader=WebSearchReader(
        max_results=3,
        search_engine="duckduckgo",
        chunk=True,
    ),
)

# Create an agent with the knowledge
agent = Agent(
    model=OpenAIChat(id="gpt-4o-mini"),
    knowledge=knowledge,
    search_knowledge=True,
    debug_mode=True,
)

# Ask the agent about the knowledge
agent.print_response(
    "What are the latest AI trends according to the search results?", markdown=True
)

​
Usage

1

Create a virtual environment

Open the Terminal and create a python virtual environment.
python3 -m venv .venv
source .venv/bin/activate
2

Install libraries

pip install -U requests beautifulsoup4 agno openai
3

Run PgVector

docker run -d \
  -e POSTGRES_DB=ai \
  -e POSTGRES_USER=ai \
  -e POSTGRES_PASSWORD=ai \
  -e PGDATA=/var/lib/postgresql/data/pgdata \
  -v pgvolume:/var/lib/postgresql/data \
  -p 5532:5432 \
  --name pgvector \
  agno/pgvector:16
4

Set environment variables

export OPENAI_API_KEY=xxx
5

Run Agent

python examples/concepts/knowledge/readers/web_search_reader.py

​
Params

ParameterTypeDefaultDescription
search_timeoutint10Timeout for search operations in seconds
request_timeoutint30Timeout for HTTP requests in seconds
delay_between_requestsfloat2.0Delay between requests in seconds
max_retriesint3Maximum number of retries for failed requests
user_agentstr"Mozilla/5.0..."User agent string for HTTP requests
search_engineLiteral["duckduckgo", "google"]"duckduckgo"Search engine to use
search_delayfloat3.0Delay between search requests in seconds
max_search_retriesint2Maximum retries for search operations
rate_limit_delayfloat5.0Delay when rate limited in seconds
exponential_backoffboolTrueWhether to use exponential backoff for retries
chunking_strategyOptional[ChunkingStrategy]SemanticChunking()Strategy for chunking content