Self-host oss-llmops-stack
Open-source LLM gateway and observability stack
- Stars
- 153
What is oss-llmops-stack?
The OSS LLMOps Stack is a reference architecture that combines two established open-source projects: LiteLLM, which acts as a unified LLM proxy/gateway providing OpenAI-compatible access to 100+ models with failover, load balancing, and cost controls; and Langfuse, which provides observability, prompt management, evaluation, and experimentation capabilities. The stack is deployed via Docker Compose or Helm charts, allowing teams to run the entire LLMOps infrastructure on their own hardware. It integrates LiteLLM callbacks with Langfuse for automatic tracing of LLM requests. The project is maintained by the Langfuse organization and is used in production by companies like Lemonade, Fletch, and Decisional.
Should you self-host oss-llmops-stack?
oss-llmops-stack scores 49/100 on selfhost.directory (how scoring works) and ranks #7 of 7 in LLM Integration. It has 153 GitHub stars.
There was no stable release in the past 12 months, so check the repository's recent activity before you rely on it.
Setup is moderate: expect some configuration beyond a single command. It is MIT-licensed.
Its closest alternative on selfhost.directory is private-gpt (81/100, 57 614 stars). It scores higher overall, so compare the two before you pick. See oss-llmops-stack vs private-gpt side by side.
Features
- Cost tracking and budgets
- Datasets and experiments
- Evaluation and experimentation
- LLM observability and tracing
- Model access controls
- Model failover and load balancing
- Open data platform with APIs and SDKs
- Prompt management and versioning
- Self-serve key management portal with SSO
- Unified LLM API
oss-llmops-stack README
More in LLM Integration
PaddleOCROpen-source, self-hosted OCR and document-parsing toolkit: scanned PDFs and photos come out as Markdown or JSON for RAG pipelines, across 100+ languages.
PangolinSelf-hosted, open-source SASE stack combining a WireGuard zero-trust VPN, identity-aware reverse proxy, PAM and an AI gateway; the AGPL Community Edition is free.
parlantOpen-source, self-hosted conversational harness for customer-facing AI agents: guidelines, journeys, canned replies and tools matched per turn by a FastAPI server.
picoclawSelf-hosted, open-source AI assistant in Go: a single binary under 10MB RAM with cron jobs, MCP tools, web search and 19+ chat channels like Telegram and Discord.
planoOpen-source, self-hosted Envoy-based AI proxy from Katanemo: LLM routing across many providers, multi-agent orchestration, guardrail filter chains and OTel observability.
pRESTOpen-source, self-hosted backend that turns a PostgreSQL or MySQL database into instant REST CRUD endpoints, custom SQL routes, JWT auth and a read-only MCP API.
private-gptOpen-source Claude-compatible API layer adding RAG with citations, tools, MCP and SQL queries to any local model server, plus a self-hosted workbench UI.
R2RSelf-hosted agentic RAG system with a RESTful API: multimodal ingestion, hybrid search, knowledge graphs, a Deep Research agent, and user & access management.
ragSelf-hosted, open-source retrieval-augmented generation chat app on txtai and Streamlit: vector RAG plus graph path queries, with your own LLM.
Resume-MatcherSelf-hosted, open-source AI resume builder that tailors a master resume to each job description with Ollama or any major LLM provider, then exports ATS-friendly PDFs.
riskkernelRiskKernel is an open-source, self-hosted runtime for AI agents that enforces spending caps, iteration limits, and time bounds with crash recovery and approval gates.
RouterlySelf-hosted LLM gateway routing requests across AI providers with multi-policy scoring, cost tracking, budget enforcement, and zero infrastructure dependencies.
0 comments