# olla

> Self-hosted LLM proxy that routes requests across inference backends with model discovery, failover, and production reliability.

- **Category:** AI
- **Licence:** Apache-2.0
- **Latest version:** v0.0.29
- **Open source:** oss
- **Languages:** Go
- **Platforms:** Docker Compose, Go, Linux, Build from Source
- **Tags:** Proxy, Lightweight, Local Ai, LLM, Inference, Nvidia, Ollama, High Performance, Load Balancer, AI, Go, Golang, Self Hosted, Amd, Self Hosted Ai, Intel, Lm studio, Litellm, Vllm, Sglang, Llm inference, Automatic failover, Llama.cpp, Llm routing, Llm proxy, Mlx, Llm router, Llama Cpp, Lmstudio

## Description

Olla is a lightweight proxy and load balancer built for LLM infrastructure. It routes requests across local inference backends like Ollama, LM Studio, vLLM, SGLang, and llama.cpp. Features include priority-based load balancing with failover, sticky sessions for conversation affinity, two proxy engines (simple and high-performance), health checks with circuit breakers, backend authentication, rate limits, and a built-in admin dashboard. The single binary uses under 50MB RAM, runs on Linux, macOS, and Windows (AMD64/ARM64), and installs via Docker, binary download, or Go. Licensed Apache-2.0.

## Links

- [Homepage](https://github.com/thushan/olla)
- [Website](https://tensorfoundry.io/products/olla)
- [Docker](ghcr.io/thushan/olla)
- [GitHub](https://github.com/thushan/olla)
- [Documentation](https://thushan.github.io/olla/)
- [Discussions](https://github.com/thushan/olla/discussions)
- [Issues](https://github.com/thushan/olla/issues)
- [Releases](https://github.com/thushan/olla/releases)

---
Source: https://selfhost.directory/project/olla