LlamaCpp | Agno

LlamaCpp runs large language models locally with efficient inference. LlamaCpp supports multiple open-source models and provides an OpenAI-compatible API server.

LlamaCpp supports a wide variety of models in GGUF format. You can find models on Hugging Face, including the default ggml-org/gpt-oss-20b-GGUF used in the examples below.

Model recommendations:

Google Gemma Models

google/gemma-2b-it-GGUF - Lightweight 2B parameter model for resource-constrained environments
google/gemma-7b-it-GGUF - Balanced 7B model for general tasks
ggml-org/gemma-3-1b-it-GGUF - Gemma 3 series, efficient for everyday use

Default Options

ggml-org/gpt-oss-20b-GGUF - Default model for general use cases
Models with different quantizations (Q4_K_M, Q8_0, etc.) for different speed/quality tradeoffs

Set up LlamaCpp

Install LlamaCpp

First, install LlamaCpp following the official installation guide:

install

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build --config Release

Or using package managers:

brew install

# macOS with Homebrew
brew install llama.cpp

Download a Model

Download a model in GGUF format following the llama.cpp model guide. For the examples below, we use ggml-org/gpt-oss-20b-GGUF.

Start the Server

In a dedicated terminal, start an installed llama.cpp server:

llama-server -hf ggml-org/gpt-oss-20b-GGUF --ctx-size 0 --jinja -ub 2048 -b 2048

For a source build, run ./build/bin/llama-server from the llama.cpp directory with the same arguments. The command downloads and loads the GGUF model and serves http://127.0.0.1:8080/v1. Keep it running. Use a second terminal in your example directory, activate your Python environment there, and run the client steps below.

Example

After starting the LlamaCpp server, use the LlamaCpp model class to access it. It uses the OpenAI client, so install the openai package:

uv pip install -U openai agno

from agno.agent import Agent
from agno.models.llama_cpp import LlamaCpp

agent = Agent(
    model=LlamaCpp(id="ggml-org/gpt-oss-20b-GGUF"),
    markdown=True
)

# Print the response in the terminal
agent.print_response("Share a 2 sentence horror story.")

Configuration

The LlamaCpp model supports customizing the server URL and model ID:

from agno.agent import Agent
from agno.models.llama_cpp import LlamaCpp

# Custom server configuration
agent = Agent(
    model=LlamaCpp(
        id="your-custom-model",
        base_url="http://localhost:8080/v1",  # Custom server URL
    ),
    markdown=True
)

View more examples here.

Params

Parameter	Type	Default	Description
`id`	`str`	`"ggml-org/gpt-oss-20b-GGUF"`	The identifier for the Llama.cpp model
`name`	`str`	`"LlamaCpp"`	The name of the model
`provider`	`str`	`"LlamaCpp"`	The provider of the model
`base_url`	`str`	`"http://127.0.0.1:8080/v1"`	The base URL for the Llama.cpp server
`api_key`	`Optional[str]`	`"not-provided"`	The API key (usually not needed for local Llama.cpp)
`temperature`	`Optional[float]`	`None`	Sampling temperature (0.0 to 2.0)
`top_p`	`Optional[float]`	`None`	Top-p sampling parameter

LlamaCpp is a subclass of the OpenAILike class and has access to the same params.

Server Configuration

The LlamaCpp server supports many configuration options:

Common Server Options

--ctx-size: Context size (0 loads the size from the model)
--batch-size, -b: Batch size for prompt processing
--ubatch-size, -ub: Physical batch size for prompt processing
--threads, -t: Number of threads to use
--host: IP address to listen on (default: 127.0.0.1)
--port: Port to listen on (default: 8080)

Model Options

--model, -m: Model file path
--hf-repo: Hugging Face model repository
--jinja: Use Jinja templating for chat formatting

For a complete list of server options, run llama-server --help (or ./build/bin/llama-server --help from a source checkout).

Performance Optimization

Hardware Acceleration

LlamaCpp supports various acceleration backends:

gpu acceleration

# NVIDIA GPU (CUDA)
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release

# Apple Metal (enabled by default on macOS)
cmake -B build
cmake --build build --config Release

# Vulkan
cmake -B build -DGGML_VULKAN=ON
cmake --build build --config Release

Model Quantization

Use quantized models for better performance:

Q4_K_M: Balanced size and quality
Q8_0: Higher quality, larger size
Q2_K: Smallest size, lower quality

Troubleshooting

Server Connection Issues

Ensure the LlamaCpp server is running and accessible:

check server

curl http://127.0.0.1:8080/v1/models

Model Loading Problems

Verify the model file exists and is in GGUF format
Check available memory for large models
Ensure the model is compatible with your LlamaCpp version

Performance Issues

Adjust batch sizes (-b, -ub) based on your hardware
Use GPU acceleration if available
Consider using quantized models for faster inference

On this page