Agent with Tools

Run a tool-enabled agent against a local vLLM server.

Code

from agno.agent import Agent
from agno.models.vllm import VLLM
from agno.tools.websearch import WebSearchTools

agent = Agent(
    model=VLLM(
        id="Qwen/Qwen2.5-7B-Instruct", top_k=20, enable_thinking=False
    ),
    tools=[WebSearchTools()],
    markdown=True,
)
agent.print_response("Whats happening in France?")

Usage

Install vLLM in a server environment using the installation guide for your hardware. The GPU commands below require a supported Linux accelerator environment. The Agno client can run in a separate Python environment with agno, openai and the example's other dependencies.

Run vllm serve in a dedicated server terminal with its environment active and keep it running. In a client terminal, activate the environment for your saved Python file, export VLLM_API_KEY, and run the agent. The examples use port 8000; for a server on another host, set the matching base_url on VLLM. A server started without --api-key accepts a placeholder key, but Agno still requires a nonempty VLLM_API_KEY.

Set up your virtual environment

uv venv --python 3.12
source .venv/bin/activate

Set the API key

export VLLM_API_KEY=xxx

Install Libraries

uv pip install -U agno openai ddgs

Start vLLM server

vllm serve Qwen/Qwen2.5-7B-Instruct \
    --enable-auto-tool-choice \
    --tool-call-parser hermes \
    --dtype float16 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.9

Run Agent

Save the code above as tool_use.py, then run:

python tool_use.py