selfhost.directory

The index of self-hostable projects — ranked, categorized and served with live updates.
Own your data and stop giving it to big tech.

koboldcpp logo

Self-host koboldcpp

v1.125

One-file GGUF LLM server with built-in chat and story UI

ReleaseOpen SourceAGPL-3.0
curl -sSL https://raw.githubusercontent.com/LostRuins/koboldcpp/concedo/android_install.sh | sh
Updated2 hours ago
LanguageC++
Setup difficultyEasy
Stars
12 359
Rating
5.0 (1)
RAM
LLAMA 7B needs at least 8GB RAM
Port
5001 (Web UI and all API endpoints)
Docker
koboldai/koboldcpp
koboldcpp screenshot 1

What is koboldcpp?

KoboldCpp is an AI text-generation server you run yourself, packaged as one standalone file and inspired by KoboldAI while building on llama.cpp. It loads GGUF (and legacy GGML) models and generates text in chat, adventure, instruct and story modes, on CPU or with CUDA, Vulkan and Metal acceleration, including optional layer offloading and an autofit mode. Media generation is built in: images (Stable Diffusion 1.5, SDXL, SD3, Flux, Qwen Image and others), video, speech recognition with Whisper, narration with several TTS engines, music with ACE-Step, plus vision input for multimodal models. Agent-style work is covered by MCP server support, tool calling, web search, TextDB retrieval-augmented generation and an integrated coding agent. On port 5001 it serves the bundled KoboldAI Lite UI and the classic llama.cpp UI together with OpenAI-compatible, KoboldAI, Ollama, A1111/Forge, ComfyUI, Whisper and XTTS endpoints, so existing clients can be repointed at it. Installation is a download-and-run binary for Windows, x86-64 Linux and Apple Silicon, an official Docker image aimed at cloud GPU rigs, and source builds for Android, OpenBSD and Raspberry Pi. Model weights are never bundled, and the project warns that koboldcpp.com is a fake site; genuine binaries come only from the GitHub releases page.

koboldcpp alternatives

All self-hosted alternatives to ChatGPT →

Should you self-host koboldcpp?

koboldcpp scores 78/100 on selfhost.directory (how scoring works) and ranks #5 of 25 in LLM Inference. It has 12 359 GitHub stars, +332 in the last 30 days.

It shipped 14 stable releases in the past year. The latest, v1.125, came out on 22 November 2026. New versions land roughly every 2 weeks.

Setup is easy: most people get it running in one sitting. An official Docker image is available. It is AGPL-3.0-licensed and written mostly in C++.

Its closest alternative on selfhost.directory is openhands (87/100, 94 006 stars). It scores higher overall, so compare the two before you pick. See koboldcpp vs openhands side by side.

Features

  • AI agent
  • Bundled web UI
  • GPU acceleration with layer offloading
  • Image Generation
  • Local GGUF model inference
  • Multi-user sharing
  • Music Generation
  • Parallel batched requests
  • Remote tunnel
  • Speech-to-Text
  • Text-to-Speech
  • Tool calling and MCP support
  • Video Generation
  • Vision / image understanding

koboldcpp README

KoboldCpp: Run local AI models with a built-in web UI

KoboldCpp is free and open-source software for running GGUF large language models (LLMs) on your own computer. Chat with an AI assistant, write stories, roleplay, or connect other apps to a local API. KoboldCpp runs on CPU or GPU and also includes text, image, video, speech and music generation with compatible models, an integrated agent, a bundled KoboldAI Lite WebUI, and many additional powerful features.

Inspired by KoboldAI and built on llama.cpp

One executable file, no installation required. Ready-to-run downloads are available for Windows, Linux, and macOS.

How to install koboldcpp

Install koboldcpp with Docker

Environment variables
KCPP_MODEL, KCPP_IMGMODEL, KCPP_MMPROJ, KCPP_ARGS
docker pull koboldai/koboldcpp

Official image is for experts only, primarily cloud GPU rentals; expects an NVIDIA or AMD GPU in an x86-64 Ubuntu environment.

How to configure koboldcpp

  • --modelModel file to load. Accepts multiple values if they are URLs (also -m, and a positional argument).

    values: GGUF/GGML file path or URL

  • --gpulayers-1Number of layers to offload to the GPU; -1 autodetects, 0 disables offload.

    values: integer

  • --contextsize8192Memory allocated for the maximum context size, in tokens (also --ctx-size, -c).

    values: 256 to 262144

  • --usecudaUse CUDA for GPU acceleration; enter a GPU ID afterwards to pick one GPU, blank to use all.

    values: GPU IDs

  • --usevulkanUse Vulkan for GPU acceleration, optionally with one or more GPU device IDs.

    values: device IDs

koboldcpp releases & version history

@ Every self-hosted release, weekly

One Monday email with the week's stable releases across the directory.

FAQ

What is grammar sampling?

Grammar sampling lets you supply a GBNF grammar that constrains the model's response to a specific syntax.

How do I see the available commands and how to use them?

Run it from a terminal with --help to print the list of available commands and launch flags.

Do I need all the files of a model, and which quantization should I pick?

You only need a single file per model; the different filenames are compression levels, with lower bit-per-weight files being smaller but slightly lower quality.

What is the difference between GGUF and GGML?

GGUF is the newer, more future-proof format and the recommended one for LLaMA models, while KoboldCpp still works with both GGUF and the older GGML versions.

Are my chats private?

Yes. It can run completely offline, sends inputs nowhere else, and the Lite UI keeps content only in the browser — though third-party clients and Horde jobs have their own privacy caveats.

What is --nomodel?

It launches the KoboldAI Lite UI by itself with no model loaded, so the UI can connect to another KoboldCpp instance or to services such as the AI Horde.

What is --config and what are .kcpps files?

A .kcpps file stores launcher preferences and settings, which can be loaded in the GUI or run directly with --config, and exported again with --exportconfig.

What is --multiuser mode?

It lets several people share one instance by queuing incoming requests and dispatching them to the right client; this is on by default, and --multiuser 0 turns it off.

How do I use streaming, and which types are supported?

Several: polled streaming is the default used by the Lite UI, pseudo-streaming is an older discouraged method, and SSE true per-token streaming is available through the API for a few third-party clients.

How do I use KoboldCpp from another device over the network?

From the same network open the host's LAN IP and port in a browser; elsewhere use the built-in Cloudflare tunnel helper (--remotetunnel), or the embedded AI Horde worker combined with the KoboldAI Lite web page.

How can I run KoboldCpp on my Android phone?

It is possible by building and running it inside Termux, though inference on the phone itself is likely slow and memory limited, so running it on another machine and connecting remotely is suggested instead.

What port does KoboldCpp use and how do I change it?

Yes, the default is 5001 and you can change it by launching with the --port parameter, then pointing your browser or API client at that port.

How much RAM or VRAM do I need to run KoboldCpp?

It depends on model size, context and quantization; with a 2048 context at Q4_0 a 3B model needs at least 4GB RAM, a 7B at least 8GB, a 13B at least 16GB and a 30B at least 32GB.

What is KoboldCpp?

It is an easy-to-use AI text-generation program for GGML and GGUF models, inspired by the original KoboldAI, built off llama.cpp and shipped as one self-contained distributable with many compatibility API endpoints and extra generation features.

Models koboldcpp runs

Popular open-weight models with koboldcpp support, with VRAM needs and licence at a glance.

openai logo20.9BA4.2BCommercial OK≈13 GB↓ 5.6M
openai logo117BA5.8BCommercial OK≈70 GB↓ 2.4M
meta-llama logo8BConditions≈6.3 GB↓ 4.1M
Qwen logo8.2BCommercial OK≈6.5 GB↓ 6.2M
zai-org logo
GLM-5.3zai-org
753BA41.8BCheck licence≈476 GB↓ 1.5M
Qwen logo14.8BCommercial OK≈11 GB↓ 13.1M
Qwen logo752MCommercial OK≈1.8 GB↓ 15.3M
Qwen logo7.6BCommercial OK≈5.5 GB↓ 6.4M

Browse all 463 models →

More in LLM Inference

Rate this project

Click to rate — no account needed.

0 comments

Posting as guest
Loading comments…