> ## Documentation Index
> Fetch the complete documentation index at: https://docs.agno.com/llms.txt
> Use this file to discover all available pages before exploring further.
# LlamaCpp
> Run local models with LlamaCpp in Agno agents.
v2.0.7
[LlamaCpp](https://github.com/ggerganov/llama.cpp) runs large language models locally with efficient inference. LlamaCpp supports multiple open-source models and provides an OpenAI-compatible API server.
LlamaCpp supports a wide variety of models in GGUF format. You can find models on Hugging Face, including the default `ggml-org/gpt-oss-20b-GGUF` used in the examples below.
Model recommendations:
### Google Gemma Models
* `google/gemma-2b-it-GGUF` - Lightweight 2B parameter model for resource-constrained environments
* `google/gemma-7b-it-GGUF` - Balanced 7B model for general tasks
* `ggml-org/gemma-3-1b-it-GGUF` - Gemma 3 series, efficient for everyday use
### Default Options
* `ggml-org/gpt-oss-20b-GGUF` - Default model for general use cases
* Models with different quantizations (Q4\_K\_M, Q8\_0, etc.) for different speed/quality tradeoffs
## Set up LlamaCpp
### Install LlamaCpp
First, install LlamaCpp following the [official installation guide](https://github.com/ggerganov/llama.cpp):
```bash install theme={null}
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build --config Release
```
Or using package managers:
```bash brew install theme={null}
# macOS with Homebrew
brew install llama.cpp
```
### Download a Model
Download a model in GGUF format following the [llama.cpp model guide](https://github.com/ggerganov/llama.cpp#obtaining-and-quantizing-models). For the examples below, we use `ggml-org/gpt-oss-20b-GGUF`.
### Start the Server
Start the LlamaCpp server with your model:
```bash start server theme={null}
llama-server -hf ggml-org/gpt-oss-20b-GGUF --ctx-size 0 --jinja -ub 2048 -b 2048
```
This starts the server at `http://127.0.0.1:8080` with OpenAI-compatible endpoints.
## Example
After starting the LlamaCpp server, use the `LlamaCpp` model class to access it. It uses the OpenAI client, so install the `openai` package:
```bash theme={null}
uv pip install -U openai agno
```
```python agent.py theme={null}
from agno.agent import Agent
from agno.models.llama_cpp import LlamaCpp
agent = Agent(
model=LlamaCpp(id="ggml-org/gpt-oss-20b-GGUF"),
markdown=True
)
# Print the response in the terminal
agent.print_response("Share a 2 sentence horror story.")
```
## Configuration
The `LlamaCpp` model supports customizing the server URL and model ID:
```python custom_config.py theme={null}
from agno.agent import Agent
from agno.models.llama_cpp import LlamaCpp
# Custom server configuration
agent = Agent(
model=LlamaCpp(
id="your-custom-model",
base_url="http://localhost:8080/v1", # Custom server URL
),
markdown=True
)
```
View more examples [here](/models/providers/local/llama-cpp/usage/basic).
## Params
| Parameter | Type | Default | Description |
| ------------- | ----------------- | ----------------------------- | ---------------------------------------------------- |
| `id` | `str` | `"ggml-org/gpt-oss-20b-GGUF"` | The identifier for the Llama.cpp model |
| `name` | `str` | `"LlamaCpp"` | The name of the model |
| `provider` | `str` | `"LlamaCpp"` | The provider of the model |
| `base_url` | `str` | `"http://127.0.0.1:8080/v1"` | The base URL for the Llama.cpp server |
| `api_key` | `Optional[str]` | `"not-provided"` | The API key (usually not needed for local Llama.cpp) |
| `temperature` | `Optional[float]` | `None` | Sampling temperature (0.0 to 2.0) |
| `top_p` | `Optional[float]` | `None` | Top-p sampling parameter |
`LlamaCpp` is a subclass of the [OpenAILike](/models/providers/openai-like) class and has access to the same params.
## Server Configuration
The LlamaCpp server supports many configuration options:
### Common Server Options
* `--ctx-size`: Context size (0 loads the size from the model)
* `--batch-size`, `-b`: Batch size for prompt processing
* `--ubatch-size`, `-ub`: Physical batch size for prompt processing
* `--threads`, `-t`: Number of threads to use
* `--host`: IP address to listen on (default: 127.0.0.1)
* `--port`: Port to listen on (default: 8080)
### Model Options
* `--model`, `-m`: Model file path
* `--hf-repo`: Hugging Face model repository
* `--jinja`: Use Jinja templating for chat formatting
For a complete list of server options, run `llama-server --help`.
## Performance Optimization
### Hardware Acceleration
LlamaCpp supports various acceleration backends:
```bash gpu acceleration theme={null}
# NVIDIA GPU (CUDA)
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release
# Apple Metal (enabled by default on macOS)
cmake -B build
cmake --build build --config Release
# Vulkan
cmake -B build -DGGML_VULKAN=ON
cmake --build build --config Release
```
### Model Quantization
Use quantized models for better performance:
* `Q4_K_M`: Balanced size and quality
* `Q8_0`: Higher quality, larger size
* `Q2_K`: Smallest size, lower quality
## Troubleshooting
### Server Connection Issues
Ensure the LlamaCpp server is running and accessible:
```bash check server theme={null}
curl http://127.0.0.1:8080/v1/models
```
### Model Loading Problems
* Verify the model file exists and is in GGUF format
* Check available memory for large models
* Ensure the model is compatible with your LlamaCpp version
### Performance Issues
* Adjust batch sizes (`-b`, `-ub`) based on your hardware
* Use GPU acceleration if available
* Consider using quantized models for faster inference