llama.cpp

by ggml-org (Georgi Gerganov)

The C/C++ inference engine that runs LLMs everywhere; the foundation of many local tools.

130.9 k stars GitHub MIT Free when localLocalOpen sourceOpen weights

View on GitHub →

What it does

GGUF format and 1.5- to 8-bit quantization
OpenAI-compatible server with a built-in web UI
Optimized for Apple Silicon, CUDA, AMD, Vulkan and CPU

Install it

Requirements: No prerequisites: ready-made binaries.

Install (macOS / Linux)

curl -LsSf https://llama.app/install.sh | sh

Install (Windows PowerShell)

irm https://llama.app/install.ps1 | iex

Start a server with a Hugging Face model

llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

The prompt for your agent

Paste it into your coding agent: it checks the requirements and installs it for you.

Claude CodeCodexCursorGemini CLIopencode

Prompt

Install llama.cpp (https://llama.app) on my machine.
1. Check the requirements: No prerequisites: ready-made binaries. If anything is missing, tell me before installing anything.
2. Follow the official method:
- Install (macOS / Linux): curl -LsSf https://llama.app/install.sh | sh
- Install (Windows PowerShell): irm https://llama.app/install.ps1 | iex
- Start a server with a Hugging Face model: llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
3. Check that it works, then explain how to launch it and where to start.

Always review what your agent is about to run, especially install commands.