vLLM

vLLM is a fast and easy-to-use library for LLM inference and serving, designed for high throughput and memory efficiency.

Prerequisites

Install vLLM and start serving a model:

The command below requires a supported Linux GPU environment. Choose the installation path for your hardware and operating system in the vLLM GPU installation guide before running it.

install vLLM

uv pip install -U agno openai vllm --torch-backend=auto

start vLLM server

vllm serve Qwen/Qwen2.5-7B-Instruct \
    --enable-auto-tool-choice \
    --tool-call-parser hermes \
    --dtype float16 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.9

This spins up the vLLM server with an OpenAI-compatible API. Set the VLLM_API_KEY environment variable. VLLM raises an error without it. If you started the server without --api-key, any value works:

set API key

export VLLM_API_KEY=xxx

VLLM connects to http://localhost:8000/v1/ by default. Override with the base_url parameter or the VLLM_BASE_URL environment variable.

Example

from agno.agent import Agent
from agno.models.vllm import VLLM

agent = Agent(
    model=VLLM(
        id="Qwen/Qwen2.5-7B-Instruct",
        base_url="http://localhost:8000/v1/",
    ),
    markdown=True
)

agent.print_response("Share a 2 sentence horror story.")

Advanced Usage

With Tools

vLLM models work with Agno tools:

with_tools.py

from agno.agent import Agent
from agno.models.vllm import VLLM
from agno.tools.hackernews import HackerNewsTools

agent = Agent(
    model=VLLM(id="Qwen/Qwen2.5-7B-Instruct"),
    tools=[HackerNewsTools()],
    markdown=True
)

agent.print_response("What's the latest news about AI?")

View more examples here.

For the full list of supported models, see the vLLM documentation.

Params

Parameter	Type	Default	Description
`id`	`str`	`"not-set"`	The id of the model to use with vLLM
`name`	`str`	`"VLLM"`	The name of the model
`provider`	`str`	`"VLLM"`	The provider of the model
`api_key`	`Optional[str]`	`None`	The API key. Falls back to the `VLLM_API_KEY` environment variable and raises an error if neither is set
`base_url`	`Optional[str]`	`None`	The base URL for the vLLM server. Falls back to the `VLLM_BASE_URL` environment variable, then `"http://localhost:8000/v1/"`
`temperature`	`float`	`0.7`	Sampling temperature
`top_p`	`float`	`0.8`	Nucleus sampling probability
`presence_penalty`	`float`	`1.5`	Penalty for repeating tokens already present
`top_k`	`Optional[int]`	`None`	Top-k sampling, sent via `extra_body`
`enable_thinking`	`Optional[bool]`	`None`	Sets `chat_template_kwargs.enable_thinking` via `extra_body`

VLLM is a subclass of the OpenAILike class and has access to the same params.

Get Started

Basics

Advanced

Production

Providers

Other

Additional Resources

Prerequisites

Example

Advanced Usage

With Tools

Params

​Prerequisites

​Example

​Advanced Usage

​With Tools

​Params

Prerequisites

Example

Advanced Usage

With Tools

Params