# LLMKube

> Open-source Kubernetes operator running LLMs on NVIDIA, AMD, Intel, and Apple Silicon with pluggable runtimes and hybrid routing.

- **Category:** AI
- **Licence:** Apache-2.0
- **Latest version:** llmkube-0.10.13
- **Open source:** oss
- **Languages:** Go
- **Platforms:** macOS, Helm, Homebrew, Build from Source
- **Tags:** Homelab, LLM, Edge Computing, Autoscaling, Inference, Nvidia, Gguf, Gpu, Kubernetes, Open Source, AI, Self Hosted, Amd, Air gapped, Multi gpu, Vllm, Apple silicon, Llama.cpp, Local llm, Operator, Mlx, Metal, Llama Cpp, Kubernetes Operator, Tgi, Model Router, Foreman, Heterogeneous Fleet

## Description

LLMKube is a Kubernetes operator for teams running LLM inference on their own mixed hardware — NVIDIA datacenter GPUs, AMD and Intel GPUs, and Apple Silicon Macs — under one control plane. It provides custom resources (Model and InferenceService) that handle model downloads, persistent caching, GPU scheduling, health checks, autoscaling with runtime-specific metrics, and OpenAI-compatible endpoints. Supported runtimes include llama.cpp for GGUF models, vLLM for HuggingFace formats, TGI, and on Apple Silicon via the Metal Agent: llama-server, mlx-server, and vllm-swift running as native macOS processes with full Metal GPU access. A ModelRouter resource adds policy-aware routing between local services and external providers like Anthropic, OpenAI, LiteLLM, Bedrock, and Vertex with strict data-egress controls for regulated workloads. Foreman, an optional add-on, dispatches coding agents across the fleet. Installation uses Helm (with an OpenShift preset), Homebrew CLI, or raw kubectl manifests. The project is Apache-2.0 licensed, actively maintained, and includes Prometheus/Grafana observability, air-gapped deployment support, and a model catalog for one-command deployments.

## Links

- [Homepage](https://github.com/defilantech/llmkube)
- [Website](https://llmkube.com)
- [Docker](ghcr.io/defilantech/llmkube-controller)
- [GitHub Repository](https://github.com/defilantech/LLMKube)
- [Documentation](https://llmkube.com/docs)
- [Discord Community](https://discord.gg/Ktz85RFHDv)
- [Blog](https://llmkube.com/blog)
- [Features](https://llmkube.com/features)
- [FAQ](https://llmkube.com/faq)
- [Getting Started Guide](https://llmkube.com/docs/getting-started)
- [ModelRouter Concept Doc](https://llmkube.com/docs/concepts/model-router)
- [Comparison with Other Projects](https://llmkube.com/docs/concepts/comparison)
- [Air-gapped Install Guide](https://llmkube.com/docs/guides/air-gapped)

---
Source: https://selfhost.directory/project/llmkube