> ## Documentation Index
> Fetch the complete documentation index at: https://docs.agno.com/llms.txt
> Use this file to discover all available pages before exploring further.

# vLLM

> Run models served by vLLM's OpenAI-compatible API in Agno agents.

[vLLM](https://docs.vllm.ai/en/latest/) is a fast and easy-to-use library for LLM inference and serving, designed for high throughput and memory efficiency.

## Prerequisites

Install vLLM and start serving a model:

```bash install vLLM theme={null}
uv pip install -U agno openai vllm
```

```bash start vLLM server theme={null}
vllm serve Qwen/Qwen2.5-7B-Instruct \
    --enable-auto-tool-choice \
    --tool-call-parser hermes \
    --dtype float16 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.9
```

This spins up the vLLM server with an OpenAI-compatible API.

Set the `VLLM_API_KEY` environment variable. `VLLM` raises an error without it. If you started the server without `--api-key`, any value works:

```bash set API key theme={null}
export VLLM_API_KEY=xxx
```

<Note>`VLLM` connects to `http://localhost:8000/v1/` by default. Override with the `base_url` parameter or the `VLLM_BASE_URL` environment variable.</Note>

## Example

<CodeGroup>
  ```python agent.py theme={null}
  from agno.agent import Agent
  from agno.models.vllm import VLLM

  agent = Agent(
      model=VLLM(
          id="Qwen/Qwen2.5-7B-Instruct",
          base_url="http://localhost:8000/v1/",
      ),
      markdown=True
  )

  agent.print_response("Share a 2 sentence horror story.")
  ```
</CodeGroup>

## Advanced Usage

### With Tools

vLLM models work with Agno tools:

```python with_tools.py theme={null}
from agno.agent import Agent
from agno.models.vllm import VLLM
from agno.tools.hackernews import HackerNewsTools

agent = Agent(
    model=VLLM(id="Qwen/Qwen2.5-7B-Instruct"),
    tools=[HackerNewsTools()],
    markdown=True
)

agent.print_response("What's the latest news about AI?")
```

<Note> View more examples [here](/models/providers/local/vllm/usage/tool-use). </Note>

For the full list of supported models, see the [vLLM documentation](https://docs.vllm.ai/en/latest/models/supported_models.html).

## Params

| Parameter          | Type             | Default     | Description                                                                                                                  |
| ------------------ | ---------------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------- |
| `id`               | `str`            | `"not-set"` | The id of the model to use with vLLM                                                                                         |
| `name`             | `str`            | `"VLLM"`    | The name of the model                                                                                                        |
| `provider`         | `str`            | `"VLLM"`    | The provider of the model                                                                                                    |
| `api_key`          | `Optional[str]`  | `None`      | The API key. Falls back to the `VLLM_API_KEY` environment variable and raises an error if neither is set                     |
| `base_url`         | `Optional[str]`  | `None`      | The base URL for the vLLM server. Falls back to the `VLLM_BASE_URL` environment variable, then `"http://localhost:8000/v1/"` |
| `temperature`      | `float`          | `0.7`       | Sampling temperature                                                                                                         |
| `top_p`            | `float`          | `0.8`       | Nucleus sampling probability                                                                                                 |
| `presence_penalty` | `float`          | `1.5`       | Penalty for repeating tokens already present                                                                                 |
| `top_k`            | `Optional[int]`  | `None`      | Top-k sampling, sent via `extra_body`                                                                                        |
| `enable_thinking`  | `Optional[bool]` | `None`      | Sets `chat_template_kwargs.enable_thinking` via `extra_body`                                                                 |

`VLLM` is a subclass of the [OpenAILike](/models/providers/openai-like) class and has access to the same params.