Self-host LLMKube
llmkube-0.10.13K8s operator for self-hosted LLM inference on mixed GPUs
- Stars
- 246
- Ports
- 8080 (OpenAI-compatible API endpoint (InferenceService)), 8081 (Controller health probes (liveness/readiness))
- Requires
- Storage class (for PVC model cache)
ghcr.io/defilantech/llmkube-controller
What is LLMKube?
LLMKube is a Kubernetes operator for teams running LLM inference on their own mixed hardware — NVIDIA datacenter GPUs, AMD and Intel GPUs, and Apple Silicon Macs — under one control plane. It provides custom resources (Model and InferenceService) that handle model downloads, persistent caching, GPU scheduling, health checks, autoscaling with runtime-specific metrics, and OpenAI-compatible endpoints. Supported runtimes include llama.cpp for GGUF models, vLLM for HuggingFace formats, TGI, and on Apple Silicon via the Metal Agent: llama-server, mlx-server, and vllm-swift running as native macOS processes with full Metal GPU access. A ModelRouter resource adds policy-aware routing between local services and external providers like Anthropic, OpenAI, LiteLLM, Bedrock, and Vertex with strict data-egress controls for regulated workloads. Foreman, an optional add-on, dispatches coding agents across the fleet. Installation uses Helm (with an OpenShift preset), Homebrew CLI, or raw kubectl manifests. The project is Apache-2.0 licensed, actively maintained, and includes Prometheus/Grafana observability, air-gapped deployment support, and a model catalog for one-command deployments.
Should you self-host LLMKube?
LLMKube scores 54/100 on selfhost.directory (how scoring works) and ranks #35 of 125 in AI. It has 246 GitHub stars.
It shipped 59 stable releases in the past year. The latest, v0.10.13, came out on 28 October 2026.
Setup is advanced: plan for several services and some ongoing maintenance. An official Docker image is available. It is Apache-2.0-licensed and written mostly in Go.
Its closest alternative on selfhost.directory is AnythingLLM (88/100, 67 666 stars). It scores higher overall, so compare the two before you pick. See LLMKube vs AnythingLLM side by side.
LLMKube README
More in LLM Inference
LumabriLumabri is a self-hosted, open-source system for running LLMs across LAN computers by distributing layers with donor approval, offering terminal and web chat.
mistral.rsmistral.rs is a self-hosted, open-source LLM inference engine in Rust serving OpenAI-compatible and Anthropic-compatible APIs with built-in web UI.
OllamaOllama is a self-hosted, open-source platform for running LLMs locally on macOS, Windows, and Linux with a REST API and official SDKs.
ollama-intel-gpuA self-hosted Docker image running Ollama on Intel Arc GPUs via IPEX-LLM, drop-in replacement for Nvidia Ollama with same API on port 11434.
production-stackSelf-hosted, open-source Helm stack that scales vLLM across a Kubernetes cluster, with KV-cache-aware request routing and a Prometheus/Grafana monitoring dashboard.
ReameSelf-hosted LLM server for CPU inference using ternary models, runs 8B models in 2.2GB RAM on free-tier ARM with OpenAI-compatible API.
Self-Hosted AI StackOpen-source Docker Compose project running local AI models, chat, documents, voice, and MCP tools on your server with optional HTTPS and GPU acceleration.
sglangSGLang is an open-source, self-hosted inference framework for LLMs and multimodal models with OpenAI-compatible API.
ShimmyA self-hosted, open-source LLM inference server: a single Rust binary that serves GGUF models through an OpenAI-compatible API, as an alternative to Ollama.
SmarterRouterSelf-hosted AI gateway that automatically routes queries to the best local or cloud LLM based on prompt analysis, with VRAM management and OpenAI-compatible API.
SNDR Core EngineSelf-hosted vLLM patch overlay running Qwen3.6 and Gemma 4 at 280K context on 24 GB GPUs with 1.5× throughput, working tool calls, and neural-graph memory.
tabbyAPITabbyAPI is a self-hosted, open-source FastAPI server providing an OpenAI-compatible REST API for running ExllamaV3-quantized LLMs on your own GPU hardware.
0 comments