> ## Documentation Index
> Fetch the complete documentation index at: https://docs.agno.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Crawl4AI

> Crawl4aiTools exposes a crawl function that fetches one or more URLs with Crawl4ai's AsyncWebCrawler and returns extracted markdown content, optionally filtered by a search query using BM25.

**Crawl4aiTools** enable an Agent to perform web crawling and scraping tasks using the Crawl4ai library.

## Prerequisites

The following example requires the `crawl4ai` library.

```shell theme={null}
uv pip install -U crawl4ai openai
```

## Example

The following agent will scrape the content from the [https://github.com/agno-agi/agno](https://github.com/agno-agi/agno) webpage:

```python theme={null}
from agno.agent import Agent
from agno.tools.crawl4ai import Crawl4aiTools

agent = Agent(tools=[Crawl4aiTools(max_length=None)])
agent.print_response("Tell me about https://github.com/agno-agi/agno.")
```

## Toolkit Params

| Parameter           | Type                       | Default              | Description                                                                               |
| ------------------- | -------------------------- | -------------------- | ----------------------------------------------------------------------------------------- |
| `max_length`        | `Optional[int]`            | `5000`               | Specifies the maximum length of the text from the webpage to be returned.                 |
| `timeout`           | `int`                      | `60`                 | Timeout in seconds for web crawling operations.                                           |
| `use_pruning`       | `bool`                     | `False`              | Enable content pruning to remove less relevant content.                                   |
| `pruning_threshold` | `float`                    | `0.48`               | Threshold for content pruning relevance scoring.                                          |
| `bm25_threshold`    | `float`                    | `1.0`                | BM25 scoring threshold for content relevance.                                             |
| `headless`          | `bool`                     | `True`               | Run browser in headless mode.                                                             |
| `wait_until`        | `str`                      | `"domcontentloaded"` | Browser wait condition before crawling (e.g., "domcontentloaded", "load", "networkidle"). |
| `proxy_config`      | `Optional[Dict[str, Any]]` | `None`               | Proxy configuration passed to the browser.                                                |
| `enable_crawl`      | `bool`                     | `True`               | Enable the web crawling functionality.                                                    |
| `all`               | `bool`                     | `False`              | Enable all available functions. When True, all enable flags are ignored.                  |

## Toolkit Functions

| Function | Description                                                                                                                        |
| -------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| `crawl`  | Crawls one or more URLs and returns the extracted content, optionally filtered by a `search_query` for relevance-based extraction. |

## Developer Resources

* [Tools](https://github.com/agno-agi/agno/blob/main/libs/agno/agno/tools/crawl4ai.py)