Spider
SpiderTools wraps the Spider API to expose search_web, scrape, and crawl functions that return web search, scrape, and crawl results.
SpiderTools connects to the hosted Spider API through spider-client to search, scrape, and crawl web content. To start using Spider, you need an API key from the Spider dashboard.
Prerequisites
The following example requires the spider-client library.
uv pip install -U agno spider-client openaiSet the provider key read by spider-client:
export SPIDER_API_KEY="your-spider-api-key"Example
The following agent will run a search query to get the latest news in USA and scrape the first search result. The agent will return the scraped data in markdown format.
from agno.agent import Agent
from agno.tools.spider import SpiderTools
agent = Agent(tools=[SpiderTools()])
agent.print_response('Can you scrape the first search result from a search on "news in USA"?', markdown=True)Toolkit Params
| Parameter | Type | Default | Description |
|---|---|---|---|
max_results | Optional[int] | None | Default maximum number of results. |
url | Optional[str] | None | Stored but unused; pass a URL to every scrape or crawl call. |
optional_params | Optional[dict] | None | Override default request options; explicit per-call search/crawl limits take precedence. |
enable_search | bool | True | Enable web search functionality. |
enable_scrape | bool | True | Enable web scraping functionality. |
enable_crawl | bool | True | Enable web crawling functionality. |
all | bool | False | Enable all tools. Overrides individual flags when True. |
Toolkit Functions
| Function | Description |
|---|---|
search_web | Searches the web for the given query. Parameters include query (str) for the search query and max_results (Optional[int], default None; uses toolkit value or 5 before optional_params overrides) for maximum results. Returns search results in JSON format. |
scrape | Scrapes the content of a webpage. Parameters include url (str) for the URL of the webpage to scrape. Returns a JSON envelope containing webpage content; return_format defaults to Markdown. |
crawl | Crawls the web starting from a URL. Parameters include url (str) for the URL to crawl and limit (Optional[int], default None; uses optional_params["limit"] or 10) for maximum pages to crawl. Returns crawl results in JSON format. |