SpiderTools is an open source web Scraper & Crawler that returns LLM-ready data. To start using Spider, you need an API key from the Spider dashboard.

​
Prerequisites

The following example requires the spider-client library. 
pip install -U spider-client

​
Example

The following agent will run a search query to get the latest news in USA and scrape the first search result. The agent will return the scraped data in markdown format.
cookbook/tools/spider_tools.py
from agno.agent import Agent
from agno.tools.spider import SpiderTools

agent = Agent(tools=[SpiderTools()])
agent.print_response('Can you scrape the first search result from a search on "news in USA"?', markdown=True)

​
Toolkit Params

ParameterTypeDefaultDescription
max_resultsOptional[int]NoneDefault maximum number of results.
urlOptional[str]NoneDefault URL for operations.
optional_paramsOptional[dict]NoneAdditional parameters for operations.
enable_searchboolTrueEnable web search functionality.
enable_scrapeboolTrueEnable web scraping functionality.
enable_crawlboolTrueEnable web crawling functionality.
allboolFalseEnable all tools. Overrides individual flags when True.

​
Toolkit Functions

FunctionDescription
searchSearches the web for the given query. Parameters include query (str) for the search query and max_results (int, default=5) for maximum results. Returns search results in JSON format.
scrapeScrapes the content of a webpage. Parameters include url (str) for the URL of the webpage to scrape. Returns markdown of the webpage.
crawlCrawls the web starting from a URL. Parameters include url (str) for the URL to crawl and limit (Optional[int], default=10) for maximum pages to crawl. Returns crawl results in JSON format.

​
Developer Resources