Self-host Ollama
v0.40.16Local-first platform for running open AI models
- Stars
- 187 003
- Rating
- 5.0 (2)
- Port
- 11434 (REST API server)
ollama/ollamaWhat is Ollama?
Ollama lets developers run open-weight LLMs like Llama, Gemma, Qwen, and DeepSeek on their own hardware. It packages model weights, configuration, and the llama.cpp inference engine into a single binary with a REST API compatible with OpenAI and Anthropic clients. Models are pulled from a public library and stored locally. The server supports GPU acceleration on NVIDIA (CUDA), AMD (ROCm), Apple (Metal), and Vulkan backends. It handles concurrent requests, model loading/unloading, context window sizing, and streaming responses. Official SDKs exist for Python and JavaScript, and a large ecosystem of community integrations connects Ollama to IDEs, chat UIs, RAG frameworks, and agent platforms. A cloud tier offers hosted models with usage-based pricing, but local inference remains free and unlimited.
Should you self-host Ollama?
Ollama scores 94/100 on selfhost.directory (how scoring works) and ranks #1 of 25 in LLM Inference. It has 187 003 GitHub stars, +4 211 in the last 30 days.
It shipped 54 stable releases in the past year. The latest, v0.40.16, came out on 8 November 2026. New versions land roughly every 2 days.
Setup is easy: most people get it running in one sitting. An official Docker image is available. It is MIT-licensed and written mostly in Go.
Its closest alternative on selfhost.directory is llama.cpp (93/100, 135 116 stars). Ollama scores higher overall. See Ollama vs llama.cpp side by side.
Features
- Cloud modelsRequires paid plan for full access; local models always free
- Concurrent model loading
- Embeddings
- GPU Acceleration
- JavaScript SDK
- Model import
- Model library
- Python SDK
- REST API
- Streaming Responses
- Structured outputs
- Tool Calling
- Vision models
FAQ
How does Ollama handle concurrent requests?
Multiple models can load if memory permits; parallel requests per model scale context and memory. Configure with OLLAMA_MAX_LOADED_MODELS, OLLAMA_NUM_PARALLEL, OLLAMA_MAX_QUEUE.
How do I keep a model loaded in memory?
Use keep_alive parameter: -1 to keep forever, duration like '10m', or set OLLAMA_KEEP_ALIVE env var. Default is 5 minutes.
Where are models stored?
macOS: ~/.ollama/models; Linux: /usr/share/ollama/.ollama/models; Windows: C:\Users\%username%\.ollama\models
How can I expose Ollama on my network?
Change the bind address with OLLAMA_HOST environment variable (default 127.0.0.1:11434).
How do I disable Ollama Cloud features?
Set disable_ollama_cloud: true in ~/.ollama/server.json or set OLLAMA_NO_CLOUD=1 environment variable.
Does Ollama send my prompts and answers back to ollama.com?
No, local runs keep data private. Cloud models process prompts but don't store or train on them.
How do I configure Ollama server?
Configure via environment variables (OLLAMA_HOST, OLLAMA_MODELS, etc.) using launchctl on macOS, systemctl on Linux, or Windows Environment Variables.
How can I tell if my model was loaded onto the GPU?
Run 'ollama ps' and check the Processor column: 100% GPU, 100% CPU, or split percentage.
How can I specify the context window size?
Set OLLAMA_CONTEXT_LENGTH environment variable (default 4096), use /set parameter num_ctx in CLI, or num_ctx in API options.
How can I view the logs?
Refer to the Troubleshooting documentation for log access instructions.
How can I upgrade Ollama?
On macOS and Windows, Ollama auto-downloads updates; click the menu/taskbar item and choose "Restart to update". On Linux, re-run the install script.
Models Ollama runs
Popular open-weight models with Ollama support, with VRAM needs and licence at a glance.
More in LLM Inference
ollama-intel-gpuA self-hosted Docker image running Ollama on Intel Arc GPUs via IPEX-LLM, drop-in replacement for Nvidia Ollama with same API on port 11434.
production-stackSelf-hosted, open-source Helm stack that scales vLLM across a Kubernetes cluster, with KV-cache-aware request routing and a Prometheus/Grafana monitoring dashboard.
ReameSelf-hosted LLM server for CPU inference using ternary models, runs 8B models in 2.2GB RAM on free-tier ARM with OpenAI-compatible API.
Self-Hosted AI StackOpen-source Docker Compose project running local AI models, chat, documents, voice, and MCP tools on your server with optional HTTPS and GPU acceleration.
sglangSGLang is an open-source, self-hosted inference framework for LLMs and multimodal models with OpenAI-compatible API.
ShimmyA self-hosted, open-source LLM inference server: a single Rust binary that serves GGUF models through an OpenAI-compatible API, as an alternative to Ollama.
SmarterRouterSelf-hosted AI gateway that automatically routes queries to the best local or cloud LLM based on prompt analysis, with VRAM management and OpenAI-compatible API.
SNDR Core EngineSelf-hosted vLLM patch overlay running Qwen3.6 and Gemma 4 at 280K context on 24 GB GPUs with 1.5× throughput, working tool calls, and neural-graph memory.
tabbyAPITabbyAPI is a self-hosted, open-source FastAPI server providing an OpenAI-compatible REST API for running ExllamaV3-quantized LLMs on your own GPU hardware.
Transformer LabSelf-hosted, open-source ML platform to train, fine-tune, evaluate and serve LLMs, diffusion and audio models locally or across GPU clusters.
Turbo-LLMSelf-hosted local LLM platform running any engine with auto-tuning, real measured tokens/sec, and OpenAI/Anthropic APIs for Claude Code.
unslothSelf-hosted, open-source UI and framework to run, fine-tune and export LLMs, GGUF/MLX and diffusion models on your own GPU or Mac.
0 comments