Llmman
Run local models through llmman's OpenAI-compatible server.
Use Llmman to connect an Agno agent to a local llmman server. The adapter defaults to qwen3:0.6b-q4_K_M at http://127.0.0.1:17434/v1 and requires no API key for local inference.
Start the model server
Follow the llmman installation instructions, then pull the example model and start the server:
llmman pull qwen3:0.6b-q4_K_M
llmman serveKeep that terminal open. In a second terminal, create and activate a virtual environment and install the Python dependencies:
uv venv .venv
source .venv/bin/activate
uv pip install -U agno openaiOn Windows, activate the environment with .venv\Scripts\Activate.ps1 in PowerShell.
Run an agent
from agno.agent import Agent
from agno.models.llmman import Llmman
agent = Agent(model=Llmman(id="qwen3:0.6b-q4_K_M"), markdown=True)
agent.print_response("Share a two-sentence horror story.")Save as agent.py and run python agent.py in the second terminal. The equivalent model string is llmman:qwen3:0.6b-q4_K_M: Agno splits at the first colon and preserves the rest of the model ID.
Configuration
| Parameter | Default | Purpose |
|---|---|---|
id | qwen3:0.6b-q4_K_M | Model reference available to the server |
base_url | http://127.0.0.1:17434/v1 | OpenAI-compatible server URL |
name | Llmman | Model name |
provider | Llmman | Provider name |
The adapter inherits OpenAI-compatible parameters. It enables the JSON-schema output path and disables native structured outputs. Tool use and output quality still depend on the selected local model.