LlamaCpp

LlamaCpp runs large language models locally with efficient inference. LlamaCpp supports multiple open-source models and provides an OpenAI-compatible API server. LlamaCpp supports a wide variety of models in GGUF format. You can find models on Hugging Face, including the default ggml-org/gpt-oss-20b-GGUF used in the examples below. Model recommendations:

Google Gemma Models

google/gemma-2b-it-GGUF - Lightweight 2B parameter model for resource-constrained environments
google/gemma-7b-it-GGUF - Balanced 7B model for general tasks
ggml-org/gemma-3-1b-it-GGUF - Gemma 3 series, efficient for everyday use

Default Options

ggml-org/gpt-oss-20b-GGUF - Default model for general use cases
Models with different quantizations (Q4_K_M, Q8_0, etc.) for different speed/quality tradeoffs

Set up LlamaCpp

Install LlamaCpp

First, install LlamaCpp following the official installation guide:

install

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build --config Release

Or using package managers:

brew install

# macOS with Homebrew
brew install llama.cpp

Download a Model

Download a model in GGUF format following the llama.cpp model guide. For the examples below, we use ggml-org/gpt-oss-20b-GGUF.

Start the Server

Start the LlamaCpp server with your model:

start server

llama-server -hf ggml-org/gpt-oss-20b-GGUF --ctx-size 0 --jinja -ub 2048 -b 2048

This starts the server at http://127.0.0.1:8080 with OpenAI-compatible endpoints.

Example

After starting the LlamaCpp server, use the LlamaCpp model class to access it. It uses the OpenAI client, so install the openai package:

uv pip install -U openai agno

from agno.agent import Agent
from agno.models.llama_cpp import LlamaCpp

agent = Agent(
    model=LlamaCpp(id="ggml-org/gpt-oss-20b-GGUF"),
    markdown=True
)

# Print the response in the terminal
agent.print_response("Share a 2 sentence horror story.")

Configuration

The LlamaCpp model supports customizing the server URL and model ID:

from agno.agent import Agent
from agno.models.llama_cpp import LlamaCpp

# Custom server configuration
agent = Agent(
    model=LlamaCpp(
        id="your-custom-model",
        base_url="http://localhost:8080/v1",  # Custom server URL
    ),
    markdown=True
)

View more examples here.

Params

Parameter	Type	Default	Description
`id`	`str`	`"ggml-org/gpt-oss-20b-GGUF"`	The identifier for the Llama.cpp model
`name`	`str`	`"LlamaCpp"`	The name of the model
`provider`	`str`	`"LlamaCpp"`	The provider of the model
`base_url`	`str`	`"http://127.0.0.1:8080/v1"`	The base URL for the Llama.cpp server
`api_key`	`Optional[str]`	`"not-provided"`	The API key (usually not needed for local Llama.cpp)
`temperature`	`Optional[float]`	`None`	Sampling temperature (0.0 to 2.0)
`top_p`	`Optional[float]`	`None`	Top-p sampling parameter

LlamaCpp is a subclass of the OpenAILike class and has access to the same params.

Server Configuration

The LlamaCpp server supports many configuration options:

Common Server Options

--ctx-size: Context size (0 loads the size from the model)
--batch-size, -b: Batch size for prompt processing
--ubatch-size, -ub: Physical batch size for prompt processing
--threads, -t: Number of threads to use
--host: IP address to listen on (default: 127.0.0.1)
--port: Port to listen on (default: 8080)

Model Options

--model, -m: Model file path
--hf-repo: Hugging Face model repository
--jinja: Use Jinja templating for chat formatting

For a complete list of server options, run llama-server --help.

Performance Optimization

Hardware Acceleration

LlamaCpp supports various acceleration backends:

gpu acceleration

# NVIDIA GPU (CUDA)
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release

# Apple Metal (enabled by default on macOS)
cmake -B build
cmake --build build --config Release

# Vulkan
cmake -B build -DGGML_VULKAN=ON
cmake --build build --config Release

Model Quantization

Use quantized models for better performance:

Q4_K_M: Balanced size and quality
Q8_0: Higher quality, larger size
Q2_K: Smallest size, lower quality

Troubleshooting

Server Connection Issues

Ensure the LlamaCpp server is running and accessible:

check server

curl http://127.0.0.1:8080/v1/models

Model Loading Problems

Verify the model file exists and is in GGUF format
Check available memory for large models
Ensure the model is compatible with your LlamaCpp version

Performance Issues

Adjust batch sizes (-b, -ub) based on your hardware
Use GPU acceleration if available
Consider using quantized models for faster inference

Get Started

Basics

Advanced

Production

Providers

Other

Additional Resources

Google Gemma Models

Default Options

Set up LlamaCpp

Install LlamaCpp

Download a Model

Start the Server

Example

Configuration

Params

Server Configuration

Common Server Options

Model Options

Performance Optimization

Hardware Acceleration

Model Quantization

Troubleshooting

Server Connection Issues

Model Loading Problems

Performance Issues

​Google Gemma Models

​Default Options

​Set up LlamaCpp

​Install LlamaCpp

​Download a Model

​Start the Server

​Example

​Configuration

​Params

​Server Configuration

​Common Server Options

​Model Options

​Performance Optimization

​Hardware Acceleration

​Model Quantization

​Troubleshooting

​Server Connection Issues

​Model Loading Problems

​Performance Issues

Google Gemma Models

Default Options

Set up LlamaCpp

Install LlamaCpp

Download a Model

Start the Server

Example

Configuration

Params

Server Configuration

Common Server Options

Model Options

Performance Optimization

Hardware Acceleration

Model Quantization

Troubleshooting

Server Connection Issues

Model Loading Problems

Performance Issues