Self-host llama.cpp
v1.0.0High-performance LLM inference engine in C/C++
- Stars
- 135 257
- Ports
- 9931 (OpenAI-compatible API server (default)), 8080 (Docker container port for llama-server)
ghcr.io/ggml-org/llama.cppWhat is llama.cpp?
llama.cpp is a high-performance LLM inference engine written in plain C/C++ with zero dependencies. It enables running quantized large language models locally on diverse hardware including Apple Silicon, x86 CPUs with AVX/AVX2/AVX512, NVIDIA GPUs via CUDA, AMD GPUs via HIP, and other accelerators. The project provides a command-line interface (llama-cli), an OpenAI-compatible API server (llama-server), and tools for model conversion and quantization. It supports 1.5-bit through 8-bit integer quantization for reduced memory usage and faster inference. The engine is built on top of the ggml tensor library and offers hybrid CPU+GPU inference for models larger than VRAM. Users can install via Docker, pre-built binaries, package managers, or build from source.
Should you self-host llama.cpp?
llama.cpp scores 93/100 on selfhost.directory (how scoring works) and ranks #2 of 25 in LLM Inference. It has 135 257 GitHub stars, +4 296 in the last 30 days.
It shipped 1742 stable releases in the past year. The latest, b12430, came out on 16 November 2026.
Setup is moderate: expect some configuration beyond a single command. An official Docker image is available. It is MIT-licensed and written mostly in C++.
Its closest alternative on selfhost.directory is Ollama (94/100, 187 151 stars). It scores higher overall, so compare the two before you pick. See llama.cpp vs Ollama side by side.
Features
- AMD GPU support
- Apple Silicon optimization
- Command Line Interface
- GBNF grammar support
- Hybrid CPU+GPU inference
- LLM inference
- Multiple backend support
- NVIDIA GPU support
- OpenAI-compatible API server
- Quantization Support
- VLM inference
- Zero Dependencies
llama.cpp README
LLM inference in C/C++
ggml / ops / maintainer PRs / dev stats / lib llama API / llama-server REST API
Models llama.cpp runs
Popular open-weight models with llama.cpp support, with VRAM needs and licence at a glance.
More in LLM Inference
llamactlSelf-hosted control plane that manages llama.cpp, MLX, and vLLM inference servers through a web dashboard with a unified OpenAI-compatible API.
llama-gptSelf-hosted, open-source ChatGPT-style chatbot that runs Llama 2 and Code Llama locally, with a web chat UI and an OpenAI-compatible API.
LLMKubeOpen-source Kubernetes operator running LLMs on NVIDIA, AMD, Intel, and Apple Silicon with pluggable runtimes and hybrid routing.
LumabriLumabri is a self-hosted, open-source system for running LLMs across LAN computers by distributing layers with donor approval, offering terminal and web chat.
mistral.rsmistral.rs is a self-hosted, open-source LLM inference engine in Rust serving OpenAI-compatible and Anthropic-compatible APIs with built-in web UI.
OllamaOllama is a self-hosted, open-source platform for running LLMs locally on macOS, Windows, and Linux with a REST API and official SDKs.
ollama-intel-gpuA self-hosted Docker image running Ollama on Intel Arc GPUs via IPEX-LLM, drop-in replacement for Nvidia Ollama with same API on port 11434.
production-stackSelf-hosted, open-source Helm stack that scales vLLM across a Kubernetes cluster, with KV-cache-aware request routing and a Prometheus/Grafana monitoring dashboard.
ReameSelf-hosted LLM server for CPU inference using ternary models, runs 8B models in 2.2GB RAM on free-tier ARM with OpenAI-compatible API.
Self-Hosted AI StackOpen-source Docker Compose project running local AI models, chat, documents, voice, and MCP tools on your server with optional HTTPS and GPU acceleration.
sglangSGLang is an open-source, self-hosted inference framework for LLMs and multimodal models with OpenAI-compatible API.
ShimmyA self-hosted, open-source LLM inference server: a single Rust binary that serves GGUF models through an OpenAI-compatible API, as an alternative to Ollama.
0 comments