Crawl4AI
Crawl4aiTools exposes a crawl function that fetches one or more URLs with Crawl4ai's AsyncWebCrawler and returns extracted Markdown content.
Crawl4aiTools enable an Agent to perform web crawling and scraping tasks using the Crawl4ai library.
Prerequisites
The following example requires the crawl4ai library.
uv pip install -U agno crawl4ai openai
crawl4ai-setupExample
The following agent will scrape the content from the https://github.com/agno-agi/agno webpage:
from agno.agent import Agent
from agno.tools.crawl4ai import Crawl4aiTools
agent = Agent(tools=[Crawl4aiTools(max_length=None)])
agent.print_response("Tell me about https://github.com/agno-agi/agno.")Toolkit Params
| Parameter | Type | Default | Description |
|---|---|---|---|
max_length | Optional[int] | 5000 | Specifies the maximum length of the text from the webpage to be returned. |
timeout | int | 60 | Page timeout in seconds, converted to page_timeout milliseconds; not a complete-run deadline. |
use_pruning | bool | False | Configure pruning; current filtered output is not returned by this adapter (see below). |
pruning_threshold | float | 0.48 | Threshold for content pruning relevance scoring. |
bm25_threshold | float | 1.0 | BM25 scoring threshold for content relevance. |
headless | bool | True | Run browser in headless mode. |
wait_until | str | "domcontentloaded" | Browser wait condition before crawling (e.g., "domcontentloaded", "load", "networkidle"). |
proxy_config | Optional[Dict[str, Any]] | None | BrowserConfig keyword arguments, e.g. {"proxy_config": {"server": "http://proxy:8080"}}. A flat server dictionary is not supported. |
enable_crawl | bool | True | Enable the web crawling functionality. |
all | bool | False | Enable all available functions. When True, all enable flags are ignored. |
Toolkit Functions
| Function | Description |
|---|---|
crawl | Crawls one or more URLs and returns the extracted content, with an optional search_query to configure BM25; see the current output limitation below. |
Current Crawl4AI stores filtered text at result.markdown.fit_markdown. This adapter checks the older top-level field, then returns raw_markdown, so pruning and BM25 do not currently produce filtered tool output. Use the current Crawl4AI API directly when you need filtered Markdown.