Self-host inference-gateway
v0.60.1Open-source LLM proxy unifying providers behind one API
- Stars
- 217
- Ports
- 8080 (Main API server), 9464 (Prometheus metrics endpoint)
ghcr.io/inference-gateway/inference-gatewayWhat is inference-gateway?
Inference Gateway is a cloud-native proxy server providing a unified OpenAI-compatible API for multiple LLM providers. It supports many providers including major cloud services and local solutions. The gateway adds Model Context Protocol (MCP) integration for automatic tool discovery, Agent-to-Agent (A2A) server for agent coordination, OPA/Rego guardrails for request filtering, OIDC authentication, and OpenTelemetry metrics. Deployment options include Docker, Docker Compose, Kubernetes with an Operator, or a standalone binary. The project is Apache-2.0 licensed with a lightweight ~10MB binary.
Should you self-host inference-gateway?
inference-gateway scores 49/100 on selfhost.directory (how scoring works) and ranks #22 of 39 in API Gateway. It has 217 GitHub stars.
It shipped 20 stable releases in the past year. The latest, v0.60.1, came out on 26 October 2026. New versions land roughly every 2 days.
Setup is moderate: expect some configuration beyond a single command. An official Docker image is available. It is Apache-2.0-licensed and written mostly in Go.
Its closest alternative on selfhost.directory is Kong Gateway (82/100, 44 314 stars). It scores higher overall, so compare the two before you pick. See inference-gateway vs Kong Gateway side by side.
Features
- Agent-to-Agent (A2A) server
- Audio APIOpenAI, ElevenLabs, and local llama-tts engine
- CLI Tool
- Function calling / Tool use
- Guardrails (OPA/Rego)
- Images APIOpenAI provider only
- Kubernetes operator
- Lightweight binary
- Model Context Protocol (MCP) integration
- OIDC Authentication
- OpenTelemetry metrics
- Privacy-first
- Streaming
- Unified OpenAI-compatible API
- Vision / MultimodalDisabled by default
inference-gateway README
More in API Gateway
IsartorA self-hosted, open-source gateway that sits between AI coding assistants and LLM providers, using local caching and small models to eliminate most repeated cloud calls.
Jentic OneJentic One is a self-hosted, open-source execution layer letting AI agents securely call real APIs without handling credentials.
Kong GatewayKong is a self-hosted, open-source API and AI Gateway for proxying, routing, load balancing, auth, and LLM/MCP traffic with 60+ AI features.
krakendSelf-hosted, open-source API gateway: a single stateless binary that aggregates, secures and transforms microservice APIs, with Docker, deb/rpm and brew installs.
llm-gatewaySelf-hosted open-source API gateway routing 250+ LLM models from 40+ providers via unified OpenAI-compatible API with cost tracking, caching, and failover.
llm-gatewaySelf-hosted OpenAI-compatible API gateway routing to OpenAI, Anthropic, and local backends with semantic routing, load balancing, and zero-trust networking via zrok.
lmgateSelf-hosted gateway adding enterprise auth, rate limits, and model ACLs to Ollama, llama.cpp, and LM Studio via Docker Compose.
LumenLumen is a self-hosted AI gateway. It provides a web chat interface and an OpenAI-compatible API proxy/
LynkrSelf-hosted, open-source LLM gateway for AI coding assistants that compresses tool tokens, caches prompts semantically, and routes by complexity across 14+ providers.
ManifestSelf-hosted, open-source LLM gateway: one OpenAI-compatible endpoint routes agents to 300+ models with provider fallbacks, cost tracking and full request logs.
mcp-context-forgeAn AI Gateway, registry, and proxy that sits in front of any MCP, A2A, or REST/gRPC APIs, exposing a unified endpoint with centralized discovery, guardrails and management. Optimizes Agent & Tool calling, and supports plugins.
mcp-gatewaySelf-hosted MCP gateway converting REST APIs into secured MCP endpoints with multi-tenant isolation, API key/OAuth auth, tool policies, and audit logging.
0 comments