Use Firecrawl with Agno to scrape and crawl the web.
firecrawl-py
library and an API key which can be obtained from Firecrawl.
Parameter | Type | Default | Description |
---|---|---|---|
api_key | str | None | API key for authentication. Uses FIRECRAWL_API_KEY env var if not provided |
enable_scrape | bool | True | Enables website scraping functionality |
enable_crawl | bool | False | Enables website crawling functionality |
enable_mapping | bool | False | Enables website mapping functionality |
enable_search | bool | False | Enables web search functionality |
all | bool | False | Enables all functionality when set to True |
formats | List[str] | None | List of formats to be used for operations |
limit | int | 10 | Maximum number of items to retrieve |
poll_interval | int | 30 | Interval in seconds between polling for results |
search_params | Dict[str, Any] | None | Parameters for search operations |
api_url | str | https://api.firecrawl.dev | API URL to use for the Firecrawl app |
Function | Description |
---|---|
scrape_website | Scrapes a website using Firecrawl. Parameters include url to specify the URL to scrape. The function supports optional formats if specified. Returns the results of the scraping in JSON format. |
crawl_website | Crawls a website using Firecrawl. Parameters include url to specify the URL to crawl, and an optional limit to define the maximum number of pages to crawl. The function supports optional formats and returns the crawling results in JSON format. |
map_website | Maps a website structure using Firecrawl. Parameters include url to specify the URL to map. Returns the mapping results in JSON format. |
search | Performs a web search using Firecrawl. Parameters include query for the search term and optional limit for maximum results. Returns search results in JSON format. |