Ollama
LLM Inference
v0.40.12
Ollama lets developers run open-weight LLMs like Llama, Gemma, Qwen, and DeepSeek on their own hardware.todayMITDockerEasy
Projects tagged #Inference
~/index>
--sort
31 projects· by score
Projects
vLLM
LLM Inference
v0.32.0vLLM is a high-performance library for running large language model inference and serving, created at UC Berkeley's Sky Computing Lab.10d agoApache-2.0DockerModerate
sglang
LLM Inference
v0.5.22SGLang is a high-performance serving framework for large language models, vision-language models, and diffusion models.9d agoApache-2.0DockerAdvanced
unsloth
Local AI
v0.1.912-betaUnsloth is an open-source toolkit and browser-based Studio for running and training AI models on your own hardware, aimed at developers and local-AI tinkerer…todayApache-2.0DockerModerate
exo
LLM Inference
v1.0.71exo is an Apache-2.0 tool from exo labs that connects all your devices into one AI cluster so you can run models too large for a single machine.4mo agoApache-2.0No DockerEasy
server
AI
v2.73.0Triton Inference Server is open-source software from NVIDIA that streamlines AI inference deployment across cloud, data center, edge, and embedded devices.27d agoBSD-3-ClauseDockerAdvanced
mistral.rs
LLM Inference
v0.9.5mistral.rs is a fast, flexible LLM inference engine built in Rust that runs locally or self-hosted.9d agoMITDockerModerate
FastChat
LLM Inference
v0.2.36FastChat is the LMSYS release repository behind the Vicuna chat models and the code that powers Chatbot Arena: an Apache-2.0 project for hosting, benchmarkin…2y agoApache-2.0No DockerDemoModerate
gpustack
LLM Inference
v2.3.0GPUStack is an open-source (Apache-2.0) manager for GPU clusters, built for serving AI models and handing out on-demand, SSH-accessible GPU instances.5d agoApache-2.0DockerModerate
envoy-ai-gateway
AI
v1.2.0Agent Router is an open-source control plane for managing AI agent traffic using Envoy Gateway technology (previously called Envoy AI Gateway).21d agoApache-2.0DockerAdvanced
production-stack
LLM Inference
vllm-stack-0.1.13vLLM Production Stack is the vLLM project's reference way to run inference clusters on Kubernetes: a Helm chart (vllm-stack) that deploys vLLM serving engine…28d agoApache-2.0DockerAdvanced
llm-gateway
API Gateway
v1.23.0An open-source API gateway connecting apps to 250+ language models from 40+ providers via one OpenAI-compatible endpoint.2d agoDockerModerate
dstack
AI
0.22.5This tool lets you manage GPU clusters across different clouds, Kubernetes, and local servers from one place.6d agoMPL-2.0DockerModerate
Transformer Lab
LLM Inference
v0.40.1Transformer Lab is an open-source machine-learning platform for researchers that pulls a fragmented AI toolchain into one interface, with a single-machine ed…14d agoAGPL-3.0No DockerModerate
GaiaNet Node
AI Agents
0.5.4GaiaNet Node is the runnable node software of the Gaia network: it turns your own machine into a decentralized AI agent service that bundles a chat model, an…1y agoGPL-3.0DockerModerate
beam
AI
gateway-0.1.837Beta9 is an open-source platform for running serverless AI workloads on your own infrastructure.1d agoAGPL-3.0No DockerAdvanced
tabbyAPI
LLM Inference
0.0.1TabbyAPI is a FastAPI-based application that serves as the official API backend for ExllamaV3, enabling self-hosted LLM inference with an OpenAI-compatible API.4mo agoAGPL-3.0DockerAdvanced
FreeLLMAPI
LLM Inference
v0.16.11FreeLLMAPI aggregates free tiers from dozens of LLM providers behind a single OpenAI-compatible endpoint.1d agoMITDocker
LLMKube
AI
llmkube-0.10.13LLMKube is a Kubernetes operator for teams running LLM inference on their own mixed hardware — NVIDIA datacenter GPUs, AMD and Intel GPUs, and Apple Silicon …todayApache-2.0DockerAdvanced
ggrun
LLM Inference
v3.2.12ggrun (from "gguf run") is a launcher for running local large language models on your own hardware.todayMITNo DockerModerate
ScribeCrate
Transcription
v2026.10.2ScribeCrate is an open-source transcription server you run on your own hardware.todayDockerDemoModerate
olla
AI
v0.0.29Olla is a lightweight proxy and load balancer built for LLM infrastructure.todayApache-2.0DockerModerate
Turbo-LLM
LLM Inference
v1.14.5TurboLLM is a self-hosted local LLM platform that runs any inference engine — llama.cpp, community forks, vLLM, MLX, KoboldCpp, llamafile, and more — auto-tu…todayFSL-1.1-ALv2No DockerModerate
AI Grid
AI
v0.3.61AI Grid (Grid) turns the computers you already own — Mac Studios, Linux GPU servers, idle workstations — into a private AI network.todayMITNo DockerModerate
hal0
AI Chat
v1.7.1hal0 is a self-hosted AI inference platform designed for AMD Strix Halo hardware but with fallbacks for other AMD GPUs, NVIDIA GPUs, and CPU-only machines.todayApache-2.0No DockerModerate
blindai
AI
v0.6.3BlindAI is an open-source AI privacy solution from Mithril Security that enables confidential AI inference using hardware-enforced Trusted Execution Environm…2y agoApache-2.0DockerDemoAdvanced
llamactl
LLM Inference
v0.22.1llamactl is a self-hosted control plane that lets you run and manage local LLM inference servers.todayMITNo DockerModerate
docker-whisper-live
AI
v2026.10.2This self-hosted transcription server handles live microphone audio and streams via WebSocket, delivering progressive text segments.3d agoDockerDemoModerate
