What is SmarterRouter?

SmarterRouter is a self-hosted AI gateway that automatically directs each prompt to the most suitable language model. It connects to local backends such as Ollama and llama.cpp as well as cloud APIs from OpenAI, Anthropic, Google, Cohere, and Mistral. The router discovers models, benchmarks them on your hardware, and learns which model handles each task type most effectively. It manages GPU memory across multiple vendors, unloading models when memory runs low. Responses are cached for instant reuse, and failed requests automatically fall back to alternative models. An OpenAI-compatible API makes it a drop-in replacement for existing clients. Admin endpoints expose model profiles, VRAM metrics, cache controls, and a dead-letter queue for failed background jobs, all protected by an API key. Configuration uses environment variables with an interactive setup wizard. Deployment options include Docker Compose with GPU-specific templates, a one-line docker-run.sh script, and manual Python installation.

Should you self-host SmarterRouter?

SmarterRouter scores 47/100 on selfhost.directory (how scoring works) and ranks #36 of 69 in AI Chat. It has 154 GitHub stars.

It shipped 10 stable releases in the past year. The latest, 2.2.5, came out on 18 April 2026. New versions land roughly every 5 days.

Setup is moderate: expect some configuration beyond a single command. An official Docker image is available. It is MIT-licensed and written mostly in Python.

Its closest alternative on selfhost.directory is LibreChat (85/100, 46 421 stars). It scores higher overall, so compare the two before you pick. See SmarterRouter vs LibreChat side by side.

Features

Admin Dashboard API
Automatic Model Profiling
External Provider Routing
Fallback & Cascading Retries
Intelligent Model Routing
Live Model Hot-Swap
LLM-as-Judge ProfilingRequires external API key
Model Filtering
Multi-backend Support
Pinned Model Support
Prometheus Metrics
Quality Preference Tuning
Semantic Response Caching
VRAM Monitoring & Management

How to configure SmarterRouter

ROUTER_ADMIN_API_KEYAuthentication key for admin endpoints (/admin/*)
Authentication key for admin endpoints (/admin/*)
ROUTER_BENCHMARK_SOURCEShuggingface,lmsysComma-separated list of benchmark data sources
Comma-separated list of benchmark data sources
values: huggingface, lmsys, artificial_analysis
ROUTER_CACHE_BACKENDmemoryCache backend implementation
Cache backend implementation
values: memory, redis
ROUTER_CACHE_ENABLEDtrueEnable smart caching of routing decisions and responses
Enable smart caching of routing decisions and responses
values: true, false
ROUTER_CACHE_MAX_SIZE500Maximum number of routing cache entries
Maximum number of routing cache entries
ROUTER_CACHE_SIMILARITY_THRESHOLD0.85Similarity threshold for semantic matching (0.0-1.0)
Similarity threshold for semantic matching (0.0-1.0)
values: 0.0-1.0
ROUTER_CACHE_TTL_SECONDS3600Time-to-live for cache entries in seconds
Time-to-live for cache entries in seconds
ROUTER_CASCADING_ENABLEDtrueIf a selected model fails, automatically retry with the next best capable model
If a selected model fails, automatically retry with the next best capable model
values: true, false
ROUTER_CORS_ALLOWED_ORIGINSCORS allowed origins
CORS allowed origins
ROUTER_DATABASE_URLsqlite:///data/router.dbDatabase connection URL
Database connection URL
ROUTER_EMBED_MODELEmbedding model for semantic similarity matching
Embedding model for semantic similarity matching
ROUTER_EXTERNAL_PROVIDERSList of enabled external providers
List of enabled external providers
values: openai, anthropic, google, cohere, mistral
ROUTER_EXTERNAL_PROVIDERS_ENABLEDfalseEnable routing to external providers
Enable routing to external providers
values: true, false
ROUTER_FEEDBACK_ENABLEDtrueCollect user feedback to improve future routing decisions
Collect user feedback to improve future routing decisions
values: true, false
ROUTER_GENERATION_TIMEOUT120Timeout for model generation requests in seconds
Timeout for model generation requests in seconds
ROUTER_JUDGE_ENABLEDfalseEnable LLM-as-Judge for quality-based profiling
Enable LLM-as-Judge for quality-based profiling
values: true, false
ROUTER_LLAMA_CPP_URLhttp://localhost:8080llama.cpp server endpoint URL
llama.cpp server endpoint URL
ROUTER_LOG_FORMATtextLog format
Log format
values: text, json
ROUTER_LOG_LEVELINFOLogging level
Logging level
values: DEBUG, INFO, WARNING, ERROR, CRITICAL
ROUTER_MODEL_FILTER_EXCLUDEComma-separated glob patterns to exclude when discovering models
Comma-separated glob patterns to exclude when discovering models
ROUTER_MODEL_FILTER_INCLUDEComma-separated glob patterns to include when discovering models
Comma-separated glob patterns to include when discovering models
ROUTER_MODEL_KEEP_ALIVE-1Controls how long models stay loaded in VRAM after each request (-1=indefinitely, 0=unload immediately, positive=seconds)
Controls how long models stay loaded in VRAM after each request (-1=indefinitely, 0=unload immediately, positive=seconds)
values: -1, 0, positive integer
ROUTER_MODEL_PREFIXString to prepend to all model names sent to the backend
String to prepend to all model names sent to the backend
ROUTER_OLLAMA_URLhttp://localhost:11434URL of your Ollama instance or OpenAI-compatible endpoint
URL of your Ollama instance or OpenAI-compatible endpoint
ROUTER_OPENAI_API_KEYAPI key for OpenAI-compatible API authentication
API key for OpenAI-compatible API authentication
ROUTER_OPENAI_BASE_URLhttps://api.openai.com/v1OpenAI-compatible API base URL
OpenAI-compatible API base URL
ROUTER_PINNED_MODELModel to keep permanently loaded in VRAM (optional)
Model to keep permanently loaded in VRAM (optional)
ROUTER_PROFILE_TIMEOUT90Base timeout for profiling operations in seconds
Base timeout for profiling operations in seconds
ROUTER_PROVIDERollamaWhich backend to use: ollama, llama.cpp, or openai
Which backend to use: ollama, llama.cpp, or openai
values: ollama, llama.cpp, openai
ROUTER_QUALITY_PREFERENCE0.5Quality vs speed tradeoff (0.0 = max speed, 1.0 = max quality)
Quality vs speed tradeoff (0.0 = max speed, 1.0 = max quality)
values: 0.0-1.0
ROUTER_RATE_LIMIT_ADMIN_REQUESTS_PER_MINUTE10Admin endpoint rate limit per client IP
Admin endpoint rate limit per client IP
ROUTER_RATE_LIMIT_CHAT_REQUESTS_PER_MINUTE100Dedicated chat endpoint rate limit per client IP
Dedicated chat endpoint rate limit per client IP
ROUTER_RATE_LIMIT_ENABLEDfalseEnable rate limiting to prevent abuse and DoS attacks
Enable rate limiting to prevent abuse and DoS attacks
values: true, false
ROUTER_RATE_LIMIT_REQUESTS_PER_MINUTE60General endpoint rate limit per client IP
General endpoint rate limit per client IP
ROUTER_REDIS_URLredis://localhost:6379Redis connection URL when using Redis cache backend
Redis connection URL when using Redis cache backend
ROUTER_REQUEST_TIMEOUT_ENABLEDtrueEnable global request timeout enforcement
Enable global request timeout enforcement
values: true, false
ROUTER_REQUEST_TIMEOUT_SECONDS300Overall request timeout budget in seconds
Overall request timeout budget in seconds
ROUTER_VRAM_AUTO_UNLOAD_ENABLEDtrueAutomatically unload unused models when VRAM pressure is high
Automatically unload unused models when VRAM pressure is high
values: true, false
ROUTER_VRAM_MAX_TOTAL_GBMaximum VRAM the router can allocate (auto-detects 90% of total if empty)
Maximum VRAM the router can allocate (auto-detects 90% of total if empty)
ROUTER_VRAM_MONITOR_ENABLEDtrueEnable VRAM monitoring with auto-detection across all GPU vendors
Enable VRAM monitoring with auto-detection across all GPU vendors
values: true, false
ROUTER_VRAM_UNLOAD_STRATEGYlruStrategy for selecting models to unload
Strategy for selecting models to unload
values: lru, largest