Web Search Reader
WebSearchReader uses DDGS to search the web and reads result pages into documents.
WebSearchReader reads data from web search results.
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
chunk | bool | True | Whether to chunk the documents |
chunk_size | int | 5000 | Size of each chunk when chunking is enabled |
encoding | Optional[str] | None | Text encoding used when reading content |
search_timeout | int | 10 | Timeout for search operations in seconds |
request_timeout | int | 30 | Timeout for HTTP requests in seconds |
delay_between_requests | float | 2.0 | Delay between requests in seconds |
max_retries | int | 3 | Maximum total attempts per result-page fetch in read(); async_read() fetches each page once |
user_agent | str | "Mozilla/5.0..." | User agent string for HTTP requests |
search_engine | Literal["duckduckgo"] | "duckduckgo" | Only accepted dispatch value. Agno calls DDGS without a backend argument, so the installed DDGS package chooses its default search backend |
search_delay | float | 3.0 | Delay between search requests in seconds |
max_search_retries | int | 2 | Maximum total search attempts, including the first, in both read methods |
rate_limit_delay | float | 5.0 | Delay when rate limited in seconds |
exponential_backoff | bool | True | Whether to use exponential backoff for retries |
chunking_strategy | Optional[ChunkingStrategy] | None | Strategy for chunking content. When None, uses SemanticChunking with the reader's chunk_size |
allowed_hosts | Optional[List[str]] | None | Hostnames the reader is allowed to fetch from. See Restricting URL Fetches. |
WebSearchReader also accepts the base Reader constructor parameters, including max_results for the number of search results to read.
Methods
read()
Searches the web for the query (up to max_results search results), reads each result page, and returns a list of documents.
| Parameter | Type | Default | Description |
|---|---|---|---|
query | str | Required | Search query to run |
async_read()
Async variant of read(). Accepts the same parameters.
Setup
Install the reader and chunking dependencies:
uv pip install -U agno ddgs beautifulsoup4 chonkie openaiThe default semantic chunker uses OpenAI embeddings and requires OPENAI_API_KEY when chunking. Set an explicit non-semantic chunking_strategy or chunk=False to avoid embedding calls; Chonkie is still required when importing the reader.
async_read() performs its search stage synchronously, which can block the event loop. Its result-page fetches are asynchronous but each has one attempt; max_retries applies only to synchronous page fetching.