Self-host Self-Hosted AI Stack
v2026.10.2Docker Compose stack for local LLMs, chat, docs, voice
- Stars
- 172
- RAM
- 8 GB
- Ports
- 3001 (AnythingLLM chat UI), 4000 (GatewayCrate (LiteLLM) AI gateway and Admin UI), 8000 (EmbedCrate (Hugging Face TEI) embeddings API), 9000 (ScribeCrate (Whisper STT) transcription API), 9090 (ScribeCrate Live (real-time STT) WebSocket), 8880 (SpeakCrate (Kokoro TTS) text-to-speech API), 3000 (UplinkCrate (MCPHub) MCP endpoint (disabled by default)), 5001 (ParseCrate (Docling) document parsing API (disabled by default)), 11434 (InferCrate (Ollama) LLM API (disabled by default))
hwdsl2/ollama-serverWhat is Self-Hosted AI Stack?
Self-Hosted AI Stack is a Docker Compose project that bundles multiple AI services into a single deployable stack. It includes InferCrate (Ollama) for running local LLM models, AnythingLLM for a password-protected web chat UI, GatewayCrate (LiteLLM) as an AI gateway with admin UI routing to 100+ providers, EmbedCrate (Hugging Face TEI) for embeddings, ScribeCrate (Whisper) for speech-to-text, ScribeCrate Live for real-time transcription, SpeakCrate (Kokoro) for text-to-speech, UplinkCrate (MCPHub) for MCP tools, and ParseCrate (Docling) for document parsing. Services auto-configure on first start with generated API keys and passwords. The stack supports lightweight variants for lower memory (as low as ~4.5 GB), optional NVIDIA GPU acceleration via CUDA, and an optional Caddy overlay for automatic HTTPS. It runs on Linux (amd64/arm64) with Docker or Podman. The configuration is MIT licensed while included services retain their own licenses.
Should you self-host Self-Hosted AI Stack?
Self-Hosted AI Stack scores 51/100 on selfhost.directory (how scoring works) and ranks #25 of 69 in AI Chat. It has 172 GitHub stars.
It shipped 3 stable releases in the past year. The latest, v2026.10.2, came out on 23 October 2026. New versions land roughly every 2 weeks.
Setup is moderate: expect some configuration beyond a single command. An official Docker image is available. It is MIT-licensed and written mostly in Shell.
Its closest alternative on selfhost.directory is openhands (87/100, 92 068 stars). It scores higher overall, so compare the two before you pick. See Self-Hosted AI Stack vs openhands side by side.
Features
- AI Gateway
- Auto-configuration
- Document ParsingDisabled by default to reduce memory usage
- Embeddings
- GPU AccelerationRequires NVIDIA GPU, driver 575.57.08+, and NVIDIA Container Toolkit
- HTTPS via CaddyRequires domain and Docker Compose 2.24.4+
- Local LLM Inference
- MCP ToolsInternal to Docker network by default; uncomment port 3000 for host access
- RAG Pipeline
- Real-time TranscriptionDisabled by default to reduce memory usage
- Speech-to-Text
- Text-to-SpeechDisabled by default to reduce memory usage
- Web Chat UI
Self-Hosted AI Stack README
More in LLM Inference
sglangSGLang is an open-source, self-hosted inference framework for LLMs and multimodal models with OpenAI-compatible API.
ShimmyA self-hosted, open-source LLM inference server: a single Rust binary that serves GGUF models through an OpenAI-compatible API, as an alternative to Ollama.
SmarterRouterSelf-hosted AI gateway that automatically routes queries to the best local or cloud LLM based on prompt analysis, with VRAM management and OpenAI-compatible API.
SNDR Core EngineSelf-hosted vLLM patch overlay running Qwen3.6 and Gemma 4 at 280K context on 24 GB GPUs with 1.5× throughput, working tool calls, and neural-graph memory.
tabbyAPITabbyAPI is a self-hosted, open-source FastAPI server providing an OpenAI-compatible REST API for running ExllamaV3-quantized LLMs on your own GPU hardware.
Transformer LabSelf-hosted, open-source ML platform to train, fine-tune, evaluate and serve LLMs, diffusion and audio models locally or across GPU clusters.
Turbo-LLMSelf-hosted local LLM platform running any engine with auto-tuning, real measured tokens/sec, and OpenAI/Anthropic APIs for Claude Code.
unslothSelf-hosted, open-source UI and framework to run, fine-tune and export LLMs, GGUF/MLX and diffusion models on your own GPU or Mac.
vLLMvLLM is a self-hosted, open-source LLM inference engine with state-of-the-art throughput via PagedAttention.
vllm-qwenSelf-hosted OpenAI-compatible inference server running Qwen3.6-27B BF16 on AMD Ryzen AI Max+ 395 with vision support, 256K context, and separated reasoning output.
WhisperLive---CPUSelf-hosted, open-source real-time Whisper transcription server with faster-whisper, TensorRT and OpenVINO backends and an OpenAI-compatible REST API.
YALSYALS is a self-hosted, open-source OpenAI-compatible API server for GGUF models via llama.cpp, with continuous batching and JSON schema support.
0 comments