Self-host LocalAI
v4.12.1Open-source AI engine for running any model on any hardware
- Stars
- 50 378
localai/localaiWhat is LocalAI?
LocalAI is an open-source AI runtime for running models locally on your own hardware. It supports language models, vision, voice, image, and video generation without requiring a GPU. The software exposes OpenAI-, Anthropic-, and ElevenLabs-compatible APIs so existing clients work with just a URL change. A small core pulls inference backends like llama.cpp, vLLM, whisper.cpp, and stable-diffusion on demand as separate container images. It runs on CPU-only, NVIDIA CUDA, AMD ROCm, Intel oneAPI, Apple Metal, Vulkan, and NVIDIA Jetson. Key features include built-in agents with tool use and RAG, realtime voice over WebRTC, distributed cluster mode with VRAM-aware routing, multi-user authentication with API keys and OIDC, per-user quotas, and a built-in web UI. You can install it via Docker, Podman, macOS DMG, Linux binaries, Kubernetes/Helm, or build from source.
Should you self-host LocalAI?
LocalAI scores 82/100 on selfhost.directory (how scoring works) and ranks #11 of 123 in AI. It has 50 378 GitHub stars, +900 in the last 30 days.
It shipped 31 stable releases in the past year. The latest, v4.12.1, came out on 23 October 2026. New versions land roughly every 1 days.
Setup is moderate: expect some configuration beyond a single command. An official Docker image is available. It is MIT-licensed and written mostly in Go.
Its closest alternative on selfhost.directory is openclaw (93/100, 394 525 stars). It scores higher overall, so compare the two before you pick. See LocalAI vs openclaw side by side.
LocalAI README
Documentation | Quick start | Models | Discord | FAQ | Blog
LocalAI is your own AI, on your own hardware, ready to work for you. It is open source. Chat with open models, run agents that use your files, and work on documents with your team. It also does voice, images, video and more. No GPU required.
LocalAI releases & version history
Patch release v4.12.1. Security fix only.
Security
- Hardened the external URL policy for user-supplied URLs (chat vision image/audio/video URLs, image and audio endpoints). Redirect targets are now validated on every hop, 100.64.0.0/10 and 0.0.0.0/8 are treated as non-public, and the connection goes to the address that was validated. See #12853. Credits to @spicysharkco for the discovery.
Full Changelog: v4.12.0...v4.12.1
AI summary Redesigned UI with streaming voice and image upscaling
🎉 LocalAI 4.12.0 Release! 🚀
LocalAI 4.12.0 is out!
This release redesigns the web UI and adds streaming voice responses, native decision routing, and more control over model memory and prompt state. It also adds image upscaling, live motion capture, Qdrant stores, and authenticated chat history. It is also a preparation for the upcoming release which will include addon support. Our plan is to move few components (e.g. Studio, Talk) to separate add-ons that can be installed on-demand from LocalAI.
Highlights:
- 🖥️ Redesigned web UI: shared controls, light and dark themes, clearer Models and Studio workflows, and node-scoped backend maintenance.
- 🎙️ Streaming voice and bundled audio models: feed LLM fragments into Qwen TTS, use explicit clause streaming with other engines, and combine audio analysis roles through Parakeet bundles.
- 🧭 Native decision routing: classify requests with decision models and use the new
/v1/decisionsadapter for predicates, choices, and scores. - 🖼️ Image upscaling in Studio: create an upscaled child image without leaving the project, with a dedicated
/v1/images/upscaleAPI. - 🛡️ Privacy and tool configuration: Responses PII filtering, opt-in rejection of uninspectable media, alias profiles, and OAuth2 client credentials for remote MCP servers.
- 🕺 Live GEM-X motion capture: stream RGB frames into a bounded, session-based pose API on CPU or Vulkan.
- 🗄️ Persistent Qdrant stores: use the existing stores API with TLS and environment-backed credentials.
- 💾 Inference controls: configure MoE expert caching, lazy tensor reads, and explicit slot prompt-cache save and restore.
The shared navigation and composer organize the web UI around models, chat, Studio, and operations.
📊 This release in numbers
| Pull requests merged | 211 |
| Commits | 307, including 96 direct commits |
| Files changed | 1,416 (+186,826 / -67,020) |
| Feature PRs | 32 |
| User-facing fix PRs | 38 |
| Model-gallery PRs | 29 |
| Top-level gallery entries | 1,660 to 1,896 (+236) |
| Dependency PRs | 90 |
Counts compare v4.11.0 with the release-preparation snapshot. Direct commits include user-facing features as well as gallery and maintenance work.
📌 TL;DR
| Area | Summary |
|---|---|
| 🖥️ Web UI | A shared visual kit, hub navigation, redesigned Models and Studio pages, per-model disk accounting, and guarded per-node backend removal. |
| 💬 Chat | Account-scoped server history, one reasoning-effort picker, editable microphone dictation, scanned-PDF image attachments, and separately sandboxed Canvas previews. |
| 🎙️ Audio | Incremental Qwen TTS, opt-in clause streaming, Parakeet bundles and speaker encoders, configurable VAD, and sound events alongside diarization. |
| 🎵 Sound | YuE2 support and optional structured expansion through a compatible native-tool chat model. YuE2 weights are non-commercial. |
| 🧭 Decisions | Native decision models can route requests; /v1/decisions adapts predicates, choices, and scores to the existing SystemOne pipeline. |
| 🛡️ Privacy and MCP | Incoming Responses text filtering, optional media rejection, protected terms and adjacent-word masking, alias profiles, and remote OAuth2 client credentials. |
| 🕺 Motion | GEM-X converts streamed RGB frames into SOMA-77 or SMPL-24 poses through a bounded duplex API. This integration is API-only. |
| 🗄️ Stores | Qdrant provides persistent vectors through /stores/*, with explicit backend selection, TLS, and chunked writes. |
| 💾 Inference | moe_cache_mib, lazy_mode, and opt-in slot save/restore add workload-specific controls without changing them into universal performance guarantees. |
| 🔧 Distributed operation | Attempt-specific load cancellation, leases, config resync, and shared root collection discovery improve the PostgreSQL/NATS architecture. |
🚀 New Features & Major Enhancements
🖥️ A shared web UI and clearer model operations
The web UI uses a shared CSS/SVG kit, ink-and-teal themes, consistent controls, and permission-aware Build and Operate hubs. Existing URLs remain available. Models exposes model details, variants, placement information, and disk usage. Studio shares a compose, job, result, and recent-results layout across its workflows.
Per-model storage accounting distinguishes shared files from model-specific files. Aggregate totals deduplicate shared bytes. A model's displayed size is not necessarily the space deleting it reclaims.
Remote manual imports now run as tracked gallery jobs. The import page and editor wait for acquisition to succeed and show failures or cancellation. Local-file imports without downloads retain synchronous behavior. Queued imports can apply recommended inference defaults, so set important parameters explicitly when reproducing a configuration.
Backends supports confirmed removal of one concrete build from one selected node. Protection checks and conflict locks prevent removal during conflicting operations. Refresh failures retain the last known inventory instead of reporting an unverified empty state.
🔗 PRs: #12526, #12551, #12587, #12599, #12785. Queued imports: background acquisition and resolver-aware queueing.
💬 Chat history, reasoning controls, and richer input
Signed-in users can synchronize conversations through /api/conversations. Account-scoped caches, persistent deletion intents, and tombstones prevent stale clients from restoring deleted conversations. This requires authentication, the WebUI enabled, and the chat_history permission. Anonymous and legacy-key-only sessions keep browser-local history; old shared browser history is not automatically assigned to an account.
One effort picker controls the per-chat reasoning_effort request. None also hides returned reasoning traces for new replies. /effort remains available; the separate /reasoning action is removed. This UI choice does not prove that every backend suppresses internal reasoning computation.
Microphone dictation appends editable text through an installed transcription model. Scanned PDFs without a text layer render their first ten pages as image attachments for a vision-capable model.
Canvas serves HTML previews with their own CSP in a sandbox without allow-same-origin. Previews expire after ten minutes and live in process memory. Allowed external HTTPS resources can load, so previews are not necessarily offline. Multi-frontend deployments need routing that reaches the process holding the preview.
Choose reasoning effort per conversation and edit dictated text before sending.
🎙️ Streaming voice responses and bundled audio analysis
Qwen TTS accepts incremental text through the optional TTSLive backend RPC. With LLM and TTS streaming enabled, one native utterance can start producing audio before the complete LLM reply arrives. Cancellation and backpressure propagate through the live path; a failed utterance does not replay after audio commitment.
pipeline: streaming: llm: true tts: true
For non-native engines, pipeline.streaming.clause_chunking: true explicitly enables bounded clause synthesis. Leaving it unset preserves whole-reply synthesis. Native Qwen input does not use this clause splitter.
The gallery adds qwen3-streaming-assistant, vits-streaming-assistant, and piper-streaming-assistant. These presets install pipeline configuration only. Install their component LLM, transcription model, VAD, and TTS model separately.
Parakeet bundles can carry multiple audio roles, including ASR, VAD, diarization, sound detection, and a speaker encoder. Available roles depend on the bundle; they do not imply one fused inference pass. Diarization can return timed sound events with include_sounds.
Speaker-equipped Parakeet models can also serve voice embedding and verification. Encoder fingerprints distinguish compatible weights, and voice stores separate different embedding dimensions. voice_verify_threshold uses cosine distance, defaults to 0.5, and accepts values between 0 and 2. This backend does not add anti-spoofing or voice-analysis support.
Realtime responses now wait for earlier audio commits to finish transcription. A waiting response ends with input_not_committed if its pending commit produces no user item, instead of starting the LLM without that turn. interrupt_response: false preserves an active response while retaining overlapping transcripts and sound events.
🔗 PRs: #12770, #12455, #12463, #12479, #12491, #12499, #12544, #12446. Direct follow-ups: Parakeet voice RPCs, ordered commit responses, and interruption control.
🎵 Structured Sound expansion and speech controls
YuE2 joins the audio.cpp sound-generation path. Ordinary Studio captions supply its style unless an explicit style parameter overrides them. YuE2 weights use CC-BY-NC-4.0 and are limited to non-commercial use.
Simple Sound can optionally use expansion_model to turn a short description into validated music fields. Final release behavior requires a llama-cpp model with use_tokenizer_template: true and native tool support. LocalAI requires exactly one strict expand_sound call; prose JSON and reasoning text are not fallback arguments. Invalid expansion stops before audio generation.
OmniVoice exposes bounded duration and postprocessing controls through speech parameters. Duration is capped at 30 seconds and quantized to generation frames. This does not guarantee an exact final WAV duration. Pocket TTS also accepts a model-load language option, such as options: ["language:french"].
🔗 PRs: #12617, #12703, #12772, #12783, #12787, #12790, #12637, #12766.
🖼️ Image upscaling and additional generation controls
POST /v1/images/upscale provides dedicated upscaling with explicit model capability checks. Studio records the output as a child image so you can continue working within the same project. Native ESRGAN mode supports tiling without loading a diffusion context. Requested scale must match the model's native scale; it is not arbitrary-scale interpolation.
custom_sigmas exposes checkpoint-specific denoising schedules. New Qwen Image assemblies use those schedules; the selected encoders make these entries text-to-image only. Their model and encoder licenses still apply.
MiniMax-H3 gains additive reference-image, reference-video, and WAV-audio fields with partition checks and bounded inputs. These fields are API-only in this release; React does not expose them. Source support does not establish end-to-end generation on every checkpoint or device.
🧭 Native decision routing and /v1/decisions
Decision models can classify requests directly for a router instead of using a general chat response as the classification protocol. The native path also supports bounded inline image decisions with a compatible format and projector.
POST /v1/decisions adapts the existing SystemOne pipeline to predicates, choices, and scores. It supports up to 64 questions and preserves typed boolean or string choices. /v1/systemone remains available.
Image input requires an image-capable decision model and inline PNG or JPEG data. Unsupported inputs fail rather than silently becoming text-only decisions. Existing Decisions permissions and usage accounting apply.
Older installed llama.cpp images may lack native Decisions support even when the server is current. Recognized stale-backend errors now return an upgrade-oriented 501 and mark the backend for upgrade. This detection does not itself rebuild an image.
🛡️ Privacy controls, model profiles, and remote MCP authentication
PII middleware now covers incoming Responses instructions, text items, function arguments, and serialized tool output. pii.block_unscanned_media: true optionally rejects media the text detectors cannot inspect. It blocks rather than OCRs or transcribes media.
pii_detection: extend_to_next_word: [ZIPCODE] extend_to_previous_word: [LASTNAME] protected_terms: [Example Supplier]
Adjacent-word extension and protected-term files add opt-in control over NER masking. Protected terms apply to NER only, not pattern detectors. They pass wherever the same term occurs, so allowlist only values safe to disclose. Extensions can over-mask.
These changes do not provide complete filtering across every request path. Agent interception and resolved historical Responses content remain gaps. Responses WebSockets with an active detector policy are refused rather than filtered; use HTTP POST for that policy.
Aliases can carry profile.system_prompt, MCP configuration, and client-tool policy while sharing one loaded target model. mcp.auto activates configured servers only within instance and user permissions. Optional numerical evidence notes are a warning heuristic, not factual verification.
Remote MCP servers support OAuth2 client_credentials, including environment-backed secrets and automatic token refresh. Token endpoints require HTTPS except explicit loopback hosts. Credential rotation through environment variables requires restarting the connection-owning process. Legacy SSE discovery remains available with constrained fallback.
🔗 PRs: #12661, #12660, #12708, #12755, #12711, #12304. Direct additions: OAuth2 credentials, protected terms, and previous-word extension.
🕺 GEM-X live motion capture
The new gemxcpp backend converts streamed RGB frames into human poses through /api/motion/sessions and a duplex WebSocket. Clients can select SOMA-77 or SMPL-24 output and use single-use, origin-bound browser tickets.
The API bounds ingress and session counts rather than accumulating an unlimited capture backlog. Limits include one session per model, four per user, and sixteen per frontend. Clients must follow flow-control credits.
This integration is API-only and supports CPU or Vulkan; macOS uses CPU, not Metal inference. It does not provide a body mesh, robot controller, or person re-identification. Session state is frontend-local. GEM-X weights carry their own NVIDIA model-license terms.
🔗 Commits: GEM-X backend and duplex API, per-user session bounds, and socket authentication hardening.
🗄️ Qdrant-backed vector stores
Use backend: qdrant-store through the existing /stores/* API to persist vectors in Qdrant. Configuration supports TLS, custom certificate authorities, and environment-backed API keys. Qdrant's gRPC default is port 6334, not REST port 6333.
Writes use chunks of at most 500 points with a timeout per chunk. Earlier chunks can remain written if a later chunk fails; a whole set is not atomic. COSINE is the default metric. Other metrics return raw scores that do not match existing cosine-based thresholds.
🔗 Commits: Qdrant backend, bounded writes, and TLS compatibility.
💾 Model memory controls and saved prompt state
options: ["moe_cache_mib:4096"]sets an opt-in GPU cache budget for host-resident MoE experts. It consumes VRAM and is workload-dependent; undersized caches can thrash.options: ["lazy_mode:on"]enables lazy reads for upstream-marked tensors.autoretains upstream selection rules;offloads them at startup. Forks without the field warn and ignore the option.options: ["slot_save_path:/var/lib/localai/slots"]enables admin slot save/restore through/v1/slots/saveand/v1/slots/restore. Saved prompt state survives model unloading. Use compatible model, quantization, and context settings; saved files are sensitive data.
Slot persistence is POSIX-only, keeps at most 32 saves per model, and does not coordinate multiple processes sharing a directory. A busy slot waits for its active request. These controls do not imply a release-wide latency or memory improvement.
🔧 Distributed recovery and opt-in diagnostics
PostgreSQL/NATS deployments gain load generations, leases, worker watchdogs, and attempt-specific cancellation through POST /api/models/:id/load-cancel. Upgrade frontends first. Model config resync defaults to 30 seconds through --model-config-resync-interval.
Root PostgreSQL collection discovery now uses a shared registry across frontends. Per-user collections are outside that change. Targeted unloading frees the named model rather than an arbitrary worker backend. These are NATS distributed-mode changes, not libp2p federation features.
Admin stop intent is not made durable by config resync: replica policies can still restart a stopped model. Do not treat resync as a persisted stop policy.
Opt-in --pprof exposes a separate numeric-loopback-only profiler. --request-phase-timing records frontend phases without adding filesystem probes. Profiles can contain sensitive data and add overhead; do not expose the profiler through public ingress. These measurements cover the Go frontend, not GPU inference on workers.
🧰 Smaller features worth knowing about
- Pause and resume your own API keys, either indefinitely or until a timestamp, without deleting them. #12521.
- Prefer eligible installed development backends with
--prefer-development-backends. Activation requires a frontend restart and separate worker configuration/restarts. #12782. - Backend OCI downloads use the configured
--download-staging-path. #12280. - Tagged releases add a thin Linux x86_64 launcher AppImage. It needs glibc 2.35+ and host graphics libraries; it does not bundle the server, backends, or models. #12647.
- Silero VAD threshold, silence, and padding settings are configurable through model options. #12430.
🐛 Bug Fixes (recap)
fix(realtime): preserve committed speech across barge-in, order transcription before response generation, and honor interruption controls. #12446 and direct follow-ups.fix(nodes): stage dedicated diarization audio on remote workers. #12474.fix(audio-cpp): forward saved voice reference transcripts. #11997.fix(sound): preserve distinct recordings after FLAC normalization. #12641.fix(chat-history): initialize the store after opening the auth database. #12788.fix(openresponses): retain synchronous server-side MCP calls and results for follow-up replay. #12711.fix(rocm): locate bundled Tensile data in architecture directories. #12411.fix(packaging): skip Linux GPU-library collection on Darwin. #12466.fix(model): remove exited backends from loaded-model listings. #12497.fix(vram): reject oversized GGUF metadata before allocation. #12560.fix(distributed): target model unloads and recover abandoned load attempts. #12524, #12537.fix(ui): repair mobile variant tables and model-import advanced controls. #12587, #12599.fix(gallery): move context options to the correct config level. Existing installed configs may need manual correction. #12590.fix(vllm-cpp): pin the MLX version expected by the engine. #12734.fix(diffusers): drop negative prompts only when negative embeddings are supplied. Commit.
🧠 Models
The gallery index grows from 1,660 to 1,896 top-level named entries (+236). This counts separate top-level variants, not nested variant references.
The Models category contains 29 PRs: 11 addition or aggregate PRs and 18 checksum-maintenance PRs. Direct gallery commits add further variants; these are change counts, not counts of distinct model architectures.
Additions include Fluid Speech Mini, multilingual Parakeet bundles with speaker support, Moondream VAD slices, LFM2.5 24B variants, MiniCPM-V 4.7 1B variants, Iris-3B, FLUX.3 Image Klein, MOSS-Image, and experimental Fish Drama 3. Runtime-backed additions elsewhere in these notes include Qwen image schedules, RealESRGAN, streaming assistant presets, and native Decisions models.
Check each model card for weight licenses, hardware requirements, and experimental limitations. A gallery entry is not proof of validation on every backend or device.
👒 Dependencies
The range includes 90 dependency PRs. Repository names below remain distinct, including llama.cpp forks.
| Source | Update PRs |
|---|---|
CrispStrobe/CrispASR |
12 |
ikawrakow/ik_llama.cpp |
12 |
0xShug0/audio.cpp |
11 |
leejet/stable-diffusion.cpp |
10 |
PrismML-Eng/llama.cpp |
8 |
ggml-org/whisper.cpp |
5 |
ggml-org/llama.cpp |
4 |
ServeurpersoCom/omnivoice.cpp |
4 |
mudler/vllm.cpp |
4 |
mudler/parakeet.cpp |
3 |
TheTom/llama-cpp-turboquant |
2 |
antirez/ds4 |
2 |
localai-org/voice-detect.cpp |
2 |
mudler/LocalRecall |
2 |
| Nine other source/platform entries | 9 |
Feature PRs also update some pins. Dependency counts alone do not describe every final engine revision.
📖 Documentation
Feature documentation covers streaming pipelines, Decisions, alias profiles, PII middleware, model imports, motion sessions, Qdrant stores, diagnostics, and backend selection. Additional docs-only PRs repair internal links, refresh news and version references, and publish the previous release post. #12786, #12484, #12458, #12442.
🙌 New Contributors
Thanks to the contributors making their first merged PRs:
- @abhishek085 in #12609.
- @aniruddh-krovvidi in #12497.
- @anton12233 in #12430.
- @drakeo338 in #12766.
- @fmterrors in #12517.
- @nexxtmobile in #12446.
Thanks also to @anush008 for the Qdrant backend and follow-up direct commits.
Full Changelog: v4.11.0...v4.12.0
AI summary Audio understanding and resilient model serving
🎉 LocalAI 4.11.0 Release! 🚀
LocalAI 4.11.0 is out!
This release makes LocalAI more useful for audio understanding, resilient model serving, structured decisions, and day-to-day operation. Audio scenes can now combine transcription, diarization, sound detection, and remembered speaker names. Ordered failover chains keep a public model available across local and remote targets, while new Studio and operations pages expose these capabilities without requiring distributed mode.
The release also adds first-class decision models through /v1/systemone, signed OCI model galleries, and Kimodo text-to-animation. It includes 243 merged pull requests, 353 commits, 79 new gallery entries, focused fixes across APIs and backends, and broad backend-source updates.
Highlights:
- 🎙️ Audio scenes and remembered speakers - combine speech-to-text, speaker diarization, sound-event detection, and speaker identification. Studio can diarize a recording, preview clean intervals, and register selected speakers by name.
- 🔀 Model failover chains - place local and remote targets behind one model name, retry before response commitment, observe health and target switches, and pin a target from the API, MCP tools, or the React UI.
- 🧭 Decision models - advertise the
decisionsuse case and answer structured choice, score, andnoulquestions through/v1/systemone, with validation and dedicated gallery models. - 📦 Signed OCI galleries - distribute complete model galleries as OCI artifacts, verify them with Sigstore policies, and use digest-bound last-known-good caches.
- 🕺 Kimodo text-to-animation - generate skeletal motion from text through
POST /3d/animate, preview it in Studio, and export binary glTF animation. - 🖥️ Operate this machine - inspect resources and locally loaded models, view logs, and stop models on a single LocalAI host without enabling distributed mode.
- 🛡️ Safer gallery and API behavior - path-confinement fixes, stricter credential checks, correct pre-stream errors, preserved streamed Responses items, and more reliable model installation.
Plus PDF attachment extraction in chat, deeper Hugging Face repository discovery, improved hardware detection, new Italian Piper voices, NeMo diarization and ASR models, and large model-gallery batches.
📸 [ screenshot: diarize a recording and remember speakers by name ]
Studio turns anonymous speaker segments into reusable named speaker profiles.
📊 This release in numbers
| Pull requests merged | 243 |
| Commits | 353 |
| Files changed | 623 (+51,035 / -2,625) |
| Development window | 15 days (2026-09-18 to 2026-10-02) |
| Human contributors | 12, of whom 4 first-time |
| Gallery entries | 1,847 to 1,926 (+79) |
Where the work landed:
| Area | Change |
|---|---|
core/ |
+23,883 / -807 across 350 files |
backend/ |
+12,437 / -186 across 119 files |
gallery/ |
+4,998 / -110 |
pkg/ |
+3,310 / -170 across 61 files |
docs/ |
+2,142 / -1,115 across 50 files |
swagger/ |
+2,254 / -100 |
📌 TL;DR
| Area | Summary |
|---|---|
| 🎙️ Audio scenes | parakeet-cpp can combine ASR, diarization, and sound detection in one model. POST /v1/audio/diarization can optionally include text and versioned speaker profiles. Transcription segments and words can carry speaker labels, and realtime sessions emit transcription-segment and sound-detection events. |
| 🗣️ Remembered speakers | Configure a compatible speaker_model to identify voices from the shared voice registry. Studio's Diarization page previews clean speaker intervals and registers a selected speaker only after an explicit Name and remember action. Speaker profiles are biometric data, remain opt-in, and are not proof of identity or consent. |
| 🔀 Failover | A model config can define an ordered failover.targets chain containing local or remote models. LocalAI retries eligible failures before committing a response, tracks health and recovery, exposes GET /api/failover plus an SSE event stream, adds admin pin and unpin controls, and reports the selected model in response headers. The localai-proxy backend connects chains to another LocalAI endpoint. |
| 🧭 Decisions | Models can explicitly declare known_usecases: [decisions] and serve structured requests through POST /v1/systemone. LocalAI validates request size, question count, state, IDs, options, levels, and noul criteria before backend invocation. The React UI exposes a Decisions capability chip and installation guidance. |
| 📦 OCI galleries | A gallery source can use oci://host/repository:tag or a digest-pinned reference. LocalAI resolves and verifies the digest, enforces layer and path limits, stages extraction atomically, and ties caches to the active verification policy. source_repository adds an exact Sigstore source constraint. |
| 🕺 Kimodo | POST /3d/animate accepts one UTF-8 prompt and generates a 60 to 150 frame skeletal animation at 30 FPS. Studio provides animation controls, preview, and local history. Output is a binary glTF .glb, not a humanoid mesh. |
| 🖥️ Single-host operations | Operate → This machine shows VRAM, RAM, CPU, models-disk usage, and running model processes. Operators can search and sort models, inspect logs, and stop a model. The existing Nodes workbench remains available when distributed mode is enabled. |
| 🧠 Models | The gallery reached 1,926 entries. Additions include large Qwen3.8 and community model batches, NeMo speech and diarization models, four Italian Piper voices, vllm-cpp structured-extraction models, and decision models including kev, Nimble, and CLM. |
🚀 New Features & Major Enhancements
🎙️ Audio scenes: speech, speakers, and sounds
parakeet-cpp can now treat audio as a scene instead of a transcript alone. One model can run speech recognition, speaker diarization, and sound-event classification, then return aligned speaker and sound information for regular and realtime requests.
POST /v1/audio/diarizationreturns diarization segments and speaker summaries.include_text=truecombines diarization with ASR.include_speaker_profiles=trueexplicitly requests versioned speaker profiles.- Transcription segments gain
speaker_name; transcript words can carry aspeakerlabel. - Realtime sessions emit
conversation.item.input_audio_transcription.segmentandconversation.item.sound_detectionevents. - Scene models can serve both
transcriptionandsound_detection, with diarization enabled in the realtime pipeline. - New gallery entries cover Nemotron 3 diarization, diarization plus ASR, CED sound models, and realtime scene variants.
Speaker profiles are sensitive biometric data. Profile export is opt-in, requires voice-recognition permission, and does not itself register a person.
🗣️ Name and remember speakers
LocalAI can match diarized segments against voices registered in its shared voice registry. A compatible parakeet-cpp configuration uses speaker_model, speaker_threshold, and speaker_margin; known voices replace anonymous labels with names while unknown voices remain SPEAKER_NN.
Studio adds a Diarization page for enrollment from ordinary recordings. It finds clean intervals for each speaker, provides audio previews, and registers a profile only when the user selects Name and remember. Profiles and recordings are not persisted in browser storage and the relevant request bodies are excluded from API trace capture.
The registry is currently process-local and ephemeral. Names disappear after restart and are not synchronized across independent frontends. Encoder identity must match exactly between enrollment and recognition.
📸 [ screenshot: speaker enrollment in Studio ]
Preview clean intervals, choose a speaker, and register the voice with an explicit action.
🔗 PR: #12414
🔀 Model failover chains and localai-proxy
One public model name can now represent an ordered chain of local and remote targets:
name: assistant-llm failover: targets: - model: preferred-local - model: remote-localai warm: true
LocalAI retries the next target only when failure happens before response commitment. Transport errors, server errors, OOM, and rate limiting can trip a target; validation errors, ordinary client errors, and client cancellation do not. Recovery probes and a minimum fallback residence period prevent rapid target flapping.
The release adds:
GET /api/failover,GET /api/failover/{chain}, andGET /api/failover/events.- Admin pin and unpin operations, also available as
list_failover_chains,pin_failover_target, andunpin_failover_targetMCP tools. X-LocalAI-Served-ModelandX-LocalAI-Failoverresponse headers.localai.model.failoverevents for realtime sessions.- A Failover Chain model template, live health strip, pin controls, Installed Models badge, and Operate → Runtime → Failover page.
- The
localai-proxyOCI backend for routing LocalAI REST capabilities to another LocalAI endpoint.
📸 [ screenshot: live failover chains and target health ]
Inspect the active target, health state, and administrative pins from the runtime page.
🔗 PR: #12285
🧭 Structured decisions and zero-shot extraction
Decision models are now a first-class LocalAI capability. A model declares known_usecases: [decisions], appears with a Decisions chip, and serves the existing SystemOne contract through POST /v1/systemone. LocalAI does not infer this use case, which keeps decision models distinct from chat and token-classification models.
The request path now enforces a 64 KiB body limit, a maximum of 64 questions, and validation for state, IDs, option counts, levels, and noul criteria. vllm-cpp carries the structured result through its unified decision ABI, and hf_overrides can merge required top-level model configuration without modifying the downloaded snapshot.
Gallery entries add Laya, GLiNER2.5-Decide, Qwen3-VL, Tev1, kev, Nimble, and CLM decision models. Existing /permute and /separate endpoints remain for NER and token classification.
📦 Signed OCI model galleries
Model galleries can now be shipped as self-contained OCI artifacts. An artifact carries its index.yaml and relative model configuration files, so a registry can distribute a complete gallery without a separate web-hosted index.
LocalAI resolves tags to digests, verifies signatures against that digest, pulls the same digest, validates artifact type and layer metadata, enforces layer-count and size bounds, confines extracted paths, and promotes the gallery cache only after complete extraction and parsing. Verification policies can require an exact source_repository URL, and cache identity includes the active verification policy.
Strict integrity mode refuses OCI galleries without a verification block. A signature or policy failure does not fall back to an older cache; transient network failures can use content previously verified under the same policy.
🕺 Kimodo text-to-animation
LocalAI adds a native kimodocpp backend and a Studio workflow for skeletal motion generation. POST /3d/animate accepts exactly one UTF-8 prompt, plus frames, steps, seed, and text_guidance. It produces a binary glTF animation at 30 FPS and reports usage through metadata.usage.
The backend has CPU and Vulkan builds, multiple model and quantization gallery entries, tracing, and usage accounting. Studio provides generation controls, animation preview, and local history. The API generates a skeleton animation rather than a humanoid mesh, and frame count is constrained to 60 through 150.
🖥️ Operate this machine
Single-node installations now have an operations page for the local host. Operate → This machine shows resource gauges for VRAM, RAM, CPU, and the models disk, then lists each running model with its backend, resident memory, CPU share, uptime, and PID.
Operators can search and sort the table, view backend logs, and stop a model after confirmation. Operate overview → Running now shows the five heaviest models, and runtime navigation includes a running-model count. In distributed mode, the route continues to show the cluster Nodes workbench.
📸 [ screenshot: Operate → This machine ]
Inspect host capacity and control every locally loaded model from one page.
🔗 PR: #12189
🧰 Smaller features worth knowing about
- Chat and Home extract text from PDF attachments before sending the request (#12374).
- Hugging Face discovery lists repositories nested more than one directory deep (#12355).
- Model capabilities report an alias target and fall back to the application default context size (#12183, #12216).
- AMD APU detection includes GTT memory, and Intel GPU probing avoids startup hangs (#12094, #12206).
- OCI gallery changes invalidate the React UI model-listing cache immediately (#12235).
- The gallery adds four Italian community Piper voices (#12121).
🐛 Bug Fixes (recap)
fix(functions)- honorfunction_arguments_keywhen building tool grammar (#11677).fix(responses)- wait for complete JSON tool calls and preserve streamed output items (#12001, #12048).fix(api)- return correct saturation and no-node status codes, expose alias targets, and report per-slot context (#12113, #12183, #12204).fix(gallery)- normalize OCI references, bind caches to verification policy, reduce repeated directory reads, and keep deletion inside the models directory (#12138, #12238, #12239, #12243, #12283, #12324, #12326).fix(modelartifacts)- reuse committed sibling files whenallow_patternsnarrows an artifact (#11484).fix(huggingface)- discover repositories nested beyond one directory level (#12355).fix(backend)- re-probe media markers after cold vision-model loading (#12254).fix(llama-cpp)- retain the default RAM cache, return pre-stream errors as errors, and let modelparallel:1override the environment (#12297, #12425, #12426).fix(whisperx)- preserve the transcript when diarization fails and report the real error (#12427).fix(distributed)- stage declared model files before loading (#12309).fix(watchdog)- ignore stale backend evictions (#12333).fix(quantization)- pin imported quantized models to the backend that produced them (#11879).fix(auth)- require validated header credentials for the CSRF exemption (#12185).fix(cosignverify)- locate bundles even when the index entry describes them incorrectly (#12165).fix(react-ui)- extract text from PDF attachments and improve text-selection contrast (#12374, #12123).fix(cloud-proxy)- surface Anthropic refusals instead of returning empty replies (#12424).fix(ollama)- report on-disk size from/api/tagsand/api/ps(#11989).fix(xsysinfo)- include AMD GTT memory and avoid Intel GPU probe hangs (#12094, #12206).
🧠 Models
- The gallery grew from 1,847 to 1,926 entries.
- A consolidated gallery batch added 293 candidate entries before deduplication and cleanup (#12124).
- A second batch added NeoHorse, Qwen3.8 Distill and Cyber variants, ByteShape, Flash Next GSQ-RCO, Occamy, Hy-MT2, Maple Preview, and more (#12221).
- NeMo speech additions cover standalone diarization, ASR, and combined diarization plus ASR (#12265).
- Four Italian community Piper voices are now available (#12121).
- vllm-cpp gallery entries add Laya, CUA-S1 forms, GLiNER2.5, kev, Nimble, and CLM decision models (#12240, #12391, #12397).
- Further additions include Hemmingway, MiMo Distill Qwen 9B, Sharp-Spark, Swift 1.5 GSQ-RCO, ThinkingCap, Agention, Qwopus Flash V2, Cyber-Tiel-Coder, and Cyber-Ornith (#12278, #12282, #12287, #12293, #12295, #12296, #12298, #12300, #12383).
👒 Dependencies
Backend sources and pinned repositories received regular updates:
ikawrakow/ik_llama.cpp: 15 updates.0xShug0/audio.cpp: 14 updates.CrispStrobe/CrispASR: 14 updates.ggml-org/llama.cpp: 12 updates.leejet/stable-diffusion.cpp: 7 updates.ServeurpersoCom/omnivoice.cpp: 7 updates.ggml-org/whisper.cpp: 6 updates.mudler/vllm.cpp: 6 updates.PrismML-Eng/llama.cpp: 6 updates.NVIDIA/NeMo-Speech.cpp: 5 updates.mudler/parakeet.cpp: 4 updates.TheTom/llama-cpp-turboquant: 3 updates.LocalAGIandlocalai-org/ced.cpp: 2 updates each.antirez/ds4,localai-org/voice-detect.cpp,PABannier/sam3.cpp, the documentation theme, nib, vllm-metal, and the vLLM CUDA wheel: 1 update each.
📖 Documentation
- Explain mixed CPU and GPU inference (#12143).
- Remove obsolete per-model documentation sections from the gallery docs (#12222).
- Clarify that upstream API keys are optional for proxy targets (#12286).
- Introduce decision models and the SystemOne API (#12428).
- Fix dead documentation and community example links (#11546, #12320, #12392).
- Correct gfx1151 environment-variable guidance (#12109).
🙌 New Contributors
Thank you to the first-time contributors in this release:
Full Changelog: v4.10.0...v4.11.0
What's Changed
Bug fixes 🐛
- fix(models): hide .tar.bz2 and .sha256 files from the model list by @localai-org-maint-bot in #12122
- fix(xsysinfo): include AMD GTT in APU VRAM detection by @leilei3167 in #12094
- fix(ui): make text selection visibly contrast the background by @blackd in #12123
- fix(gallery): strip oci:// before upgrade-check digest lookup by @mudler-agent in #12138
- fix: preserve vllm-omni imports after backend relocation by @mudler-agent in #12137
- fix: correct stale symbol names in doc comments by @mudler-agent in #12139
- fix(cosignverify): find a bundle the index entry describes badly by @localai-org-maint-bot in #12165
- fix(api): report an alias's target in /v1/models/capabilities by @localai-org-maint-bot in #12183
- fix(make): fail the protoc download on an HTTP error by @localai-org-maint-bot in #12187
- fix(auth): require validated header credentials for CSRF exemption by @richiejp in #12185
- fix(openai): return HTTP errors for pre-stream failures and report per-slot context by @mudler-agent in #12204
- fix(xsysinfo): avoid startup hang on intel_gpu_top by @leilei3167 in #12206
- fix(sglang): support msgspec-based ServerArgs (sglang >= 0.5.20) by @pos-ei-don in #12155
- fix(vllm-omni): remove invalid syntax in test.py by @H-XX-D in #12135
- fix(turboquant): extend D512 flash-attn patch to all turbo V types by @mudler-agent in #12234
- fix(gallery): strip the oci:// scheme before every registry lookup by @mudler-agent in #12238
- fix(gallery): tie oci:// gallery caches to the verification policy by @mudler-agent in #12239
- fix(gallery): verification follow-ups for oci:// galleries by @mudler-agent in #12243
- fix(backend): re-probe MediaMarker after cold vision model load by @leilei3167 in #12254
- fix(gallery): read the models dir once per gallery listing by @localai-org-maint-bot in #12283
- fix(swagger): describe backend metadata as an object by @localai-org-maint-bot in #12178
- fix(compose): request NVIDIA compute capability by @localai-org-maint-bot in #11990
- fix(modelartifacts): reuse committed sibling files for narrowed allow_patterns by @SuperMarioYL in #11484
- fix(responses): wait for complete JSON tool calls by @localai-org-maint-bot in #12001
- fix(responses): preserve streamed output items by @localai-org-maint-bot in #12048
- fix(kokoros): add missing animate3_d stub to Backend trait impl by @mudler-agent in #12301
- fix(ollama): report on-disk size for /api/tags and /api/ps by @leilei3167 in #11989
- fix(quantization): pin the producing backend on imported quantized models (#11875) by @Anai-Guo in #11879
- fix: return correct HTTP status codes for saturation and no-nodes-available by @localai-org-maint-bot in #12113
- [router] fix: re-seed the knn corpus index when the vector store comes back empty by @walcz-de in #12267
- fix(functions): honor function_arguments_key when building the tool grammar by @Anai-Guo in #11677
- fix(models): fallback to application config default context size in /v1/models/capabilities (#12202) by @PINYOPATTANAWASANPORN in #12216
- fix: point docker-compose default at gallery phi-2-chat by @leilei3167 in #11987
- fix(distributed): stage the files a model install declares by @mudler-agent in #12309
- fix(gallery): keep model deletion inside the models directory by @mudler-agent in #12324
- fix: make the remaining VerifyPath checks effective by @mudler-agent in #12326
- fix(llama-cpp): keep llama.cpp's default cache_ram instead of no limit by @walcz-de in #12297
- fix(huggingface): list repos nested more than one directory deep by @mudler-agent in #12355
- fix(react-ui): extract text from PDF attachments in chat and home by @mudler-agent in #12374
- fix(vllm-cpp): annotate the hf_overrides config.json read for gosec by @mudler-agent in #12380
- fix(watchdog): ignore stale backend evictions by @localai-org-maint-bot in #12333
- fix(funasr): select Python 3.12 tokenizers by @mudler-agent in #12402
- fix(llama-cpp): let parallel:1 in the model options win over LLAMACPP_PARALLEL by @walcz-de in #12426
- fix(whisperx): keep the transcript when diarization fails, report real errors by @walcz-de in #12427
- fix(cloud-proxy): surface Anthropic refusals instead of empty replies by @walcz-de in #12424
- fix(llama-cpp): do not stream the error text as content on pre-stream failures by @walcz-de in #12425
Exciting New Features 🎉
- feat: Add kimodo.cpp and 3D animation API/UI by @richiejp in #12095
- feat(vllm-cpp): add video_lora_dir option for runtime prompt-activated LoRA by @localai-org-maint-bot in #12119
- feat(openai): add negative_prompt field to image generation request by @lqp in #12031
- feat(swagger): update swagger by @localai-org-maint-bot in #12000
- feat(swagger): update swagger by @localai-org-maint-bot in #12148
- feat(kimodocpp): track usage through generic backend metadata by @richiejp in #12162
- feat(ui): show running models and host gauges on single-node installs by @localai-org-maint-bot in #12189
- feat(stablediffusion-ggml): Qwen-Image 2.1 support + gallery GGUF by @localai-org-maint-bot in #12190
- feat(kimodo): Add observability hooks by @richiejp in #12184
- feat(audio-cpp): AUDIOCPP_DEFAULT_BACKEND fallback for models without a backend option by @blackd in #12133
- feat(vllm-cpp): add GLiNER2.5 NER via TokenClassify by @mudler-agent in #12140
- feat(vllm-cpp): unify decision pipeline through Score RPC with vllm_decide ABI v29 by @mudler-agent in #12247
- feat(system): report per-model DRM VRAM by @localai-org-maint-bot in #12026
- sglang backend: pass through thinking_budget + require_reasoning by @pos-ei-don in #12193
- feat(swagger): update swagger by @localai-org-maint-bot in #12308
- feat(distributed): report worker version and show models in node inspector by @mudler-agent in #12328
- feat(failover): serve a model name from a chain of local and remote targets by @mudler-agent in #12285
- feat(parakeet-cpp): speaker diarization, sound detection and live scene events by @localai-org-maint-bot in #12335
- feat: decisions usecase for decision models, with gallery tagging by @mudler-agent in #12373
- feat(parakeet-cpp): name speakers from the shared voice registry by @localai-org-maint-bot in #12382
- feat(audio): remember speakers from diarization by @mudler-agent in #12414
🧠 Models
- feat(gallery): add four Italian community Piper voices by @localai-org-maint-bot in #12121
- feat(gallery): consolidate 25 pending gallery PRs by @mudler-agent in #12124
- feat(gallery): galleries published as OCI artifacts by @localai-org-maint-bot in #12167
- batch(gallery): merge 14 gallery model-addition PRs by @mudler-agent in #12221
- feat(gallery): optionally pin the signing certificate's source repository by @mudler-agent in #12235
- feat(gallery): add vllm-cpp entries for laya, cua-s1-forms, and gliner2.5 by @mudler-agent in #12240
- feat(gallery): add nemo-speech-cpp diarization and ASR models by @mudler-agent in #12265
- feat(gallery): read metadata for system-path backends, enabling variant aliases by @blackd in #12141
- chore(gallery): add Hemmingway and remove invalid chat entry by @localai-org-maint-bot in #12278
- chore(gallery): add MiMo distill Qwen 9B variants by @localai-org-maint-bot in #12282
- feat(gallery): publish signed OCI fallbacks by @localai-org-maint-bot in #12182
- chore(gallery): add Sharp-Spark 4B variants by @localai-org-maint-bot in #12287
- chore(gallery): add Swift 1.5 GSQ-RCO variants by @localai-org-maint-bot in #12293
- chore(gallery): add ThinkingCap Qwen3.8 variants by @localai-org-maint-bot in #12295
- chore(gallery): add Agention Qwen3.8 variants by @localai-org-maint-bot in #12296
- chore(gallery): add Qwopus Flash V2 variants by @localai-org-maint-bot in #12298
- chore(gallery): add Cyber-Tiel-Coder variants by @localai-org-maint-bot in #12300
- feat(gallery): add kev-0.8b on vllm-cpp as a decisions model by @mudler-agent in #12391
- chore(gallery): add Cyber-Ornith 1.5 variants by @localai-org-maint-bot in #12383
- feat(gallery): add Nimble 9B and CLM decision models, bump vllm.cpp to a19294a9 by @mudler-agent in #12397
📖 Documentation and examples
- docs: ⬆️ update docs version mudler/LocalAI by @localai-org-maint-bot in #12111
- docs: explain mixed CPU/GPU inference by @localai-org-maint-bot in #12143
- docs(gpu): drop false auto-set claim for gfx1151 env vars by @leilei3167 in #12109
- docs(gallery): remove per-model documentation sections by @mudler-agent in #12222
- docs(proxy): clarify optional upstream API keys by @localai-org-maint-bot in #12286
- docs: replace dead chatbot-ui example link with repo root by @yzxcj797 in #11546
- docs: fix 11 dead links in the documentation by @pratikgx in #12320
- docs: fix Discord, Slack and Telegram example links by @pratikgx in #12392
- docs: introduce decision models on the blog by @localai-org-maint-bot in #12428
👒 Dependencies
- chore: ⬆️ Update ServeurpersoCom/omnivoice.cpp to
c2257c833333f222d64dc9d437afdcece33ceb0bby @localai-org-maint-bot in #12116 - chore: ⬆️ Update NVIDIA/NeMo-Speech.cpp to
07003daa7eefea542076310722ccaa89709ee3c3by @localai-org-maint-bot in #12115 - chore: ⬆️ Update ggml-org/llama.cpp to
972d2313bc0bf0a45f634f77d95c9fb03aeab12cby @localai-org-maint-bot in #12090 - chore: ⬆️ Update PABannier/sam3.cpp to
416186c501d060df7ca02989d49b38080f5f81f3by @localai-org-maint-bot in #12091 - chore: ⬆️ Update mudler/vllm.cpp to
e27e6d1c8f9ccd2803d37030f8a677507fe6e314by @localai-org-maint-bot in #12089 - chore: ⬆️ Update leejet/stable-diffusion.cpp to
cc515a01f9d0e3f6b975234cc934b807f55bcd35by @localai-org-maint-bot in #12087 - chore: ⬆️ Update ikawrakow/ik_llama.cpp to
dc31024448b8f18eac0cd5c2e200b6c7e015ef7aby @localai-org-maint-bot in #12088 - chore: ⬆️ Update 0xShug0/audio.cpp to
f2b4937306daa25f5c78520f3c626ed31495a37aby @localai-org-maint-bot in #12086 - chore(deps): bump docs/themes/hugo-theme-relearn from
8bb66fatoaa16cb1by @dependabot[bot] in #12096 - chore(deps): bump actions/checkout from 6 to 7 by @dependabot[bot] in #12108
- chore(deps): bump protobuf from 7.35.0 to 7.36.1 in /backend/python/transformers by @dependabot[bot] in #12098
- chore(deps): bump grpcio from 1.83.1 to 1.84.0 in /backend/python/vllm by @dependabot[bot] in #12099
- chore(deps): bump grpcio from 1.83.1 to 1.84.0 in /backend/python/common/template by @dependabot[bot] in #12101
- chore(deps): bump grpcio from 1.83.1 to 1.84.0 in /backend/python/rerankers by @dependabot[bot] in #12104
- chore: ⬆️ Update ServeurpersoCom/omnivoice.cpp to
cd6922ac3cb465f1c0a22465e77db21d367204feby @localai-org-maint-bot in #12126 - chore: ⬆️ Update mudler/vllm.cpp to
f3cd97e379fbeca4e50415edbdd52d2517b98ef8by @localai-org-maint-bot in #12132 - chore: ⬆️ Update ikawrakow/ik_llama.cpp to
2ae132fa601ea06818ed3584f50f7eb4f72d4967by @localai-org-maint-bot in #12131 - chore: ⬆️ Update ggml-org/whisper.cpp to
5670d5c0bbcb148feabef84400a07cfca9aa3b30by @localai-org-maint-bot in #12130 - chore: ⬆️ Update ggml-org/llama.cpp to
50631b3d2c569ad8e5c112090cd28570b1268ee0by @localai-org-maint-bot in #12129 - chore: ⬆️ Update leejet/stable-diffusion.cpp to
2ea8aff7ef603977dc2ece7856bf9736dba96652by @localai-org-maint-bot in #12127 - chore: ⬆️ Update 0xShug0/audio.cpp to
a074d6b8cdb16b89cd028876e83629a538d49b9aby @localai-org-maint-bot in #12125 - chore: ⬆️ Update CrispStrobe/CrispASR to
647db2c7abed1fc82a69767f6e8b3993b94b8417by @localai-org-maint-bot in #12112 - chore: ⬆️ Update CrispStrobe/CrispASR to
7bd1d6062eb3d96dfb68fb240f8736399ba490c3by @localai-org-maint-bot in #12153 - chore: ⬆️ Update leejet/stable-diffusion.cpp to
1330cebae8f2ba99249df846cc0c9444fcbd4308by @localai-org-maint-bot in #12152 - chore: ⬆️ Update ikawrakow/ik_llama.cpp to
401a09d2f534d2eeabb0a37919ebc5a2cbc56ac6by @localai-org-maint-bot in #12151 - chore: ⬆️ Update mudler/vllm.cpp to
ea8c83d75f461a520e41328c44bde6c949453fa6by @localai-org-maint-bot in #12149 - chore: ⬆️ Update 0xShug0/audio.cpp to
a7b58a6d3d6ae4143c485266b1c6c09898ad8c72by @localai-org-maint-bot in #12150 - chore: ⬆️ Update PrismML-Eng/llama.cpp to
9a9394a895b96003ca842a6041cb28ac49a108f7by @localai-org-maint-bot in #12114 - chore: ⬆️ Update ggml-org/llama.cpp to
e613ef2c81bae98d59850d061ac29e6e3e88cb00by @localai-org-maint-bot in #12157 - chore: ⬆️ Update antirez/ds4 to
0aaea5a238fb41a35106a551e73c8409dfb751acby @localai-org-maint-bot in #12168 - chore: ⬆️ Update ikawrakow/ik_llama.cpp to
9cba2e3874df6f598fd339c4c6c7d5fc2b44645bby @localai-org-maint-bot in #12174 - chore: ⬆️ Update CrispStrobe/CrispASR to
46612927d8ed7a98e88fb9f411768a2ccbbe1170by @localai-org-maint-bot in #12171 - chore: ⬆️ Update ServeurpersoCom/omnivoice.cpp to
ae9dd24b6a5ff72bf50519490092ad14c64627a8by @localai-org-maint-bot in #12169 - chore: ⬆️ Update 0xShug0/audio.cpp to
e3de8e3f3cbfac55ffa58df71426c41550a8598bby @localai-org-maint-bot in #12176 - chore: ⬆️ Update ggml-org/llama.cpp to
ce8caa6e60a03093351d6016a818720e0d46f0fbby @localai-org-maint-bot in #12177 - chore: ⬆️ Update 0xShug0/audio.cpp to
17cc8980e9c8f8073796aead8a91c809511cbab1by @localai-org-maint-bot in #12199 - chore: ⬆️ Update CrispStrobe/CrispASR to
5cfdc754c04d7bb3f0ab637f0e09502eda22d1aeby @localai-org-maint-bot in #12198 - chore: ⬆️ Update NVIDIA/NeMo-Speech.cpp to
302ebc93f096d03395e5d86c643897b8bf0fd9a8by @localai-org-maint-bot in #12196 - chore: ⬆️ Update PrismML-Eng/llama.cpp to
01ae597e3f7d4742909e1e831abb12fe3d24b2cfby @localai-org-maint-bot in #12195 - chore: ⬆️ Update ikawrakow/ik_llama.cpp to
baac291dc9d531927760b48451d8dfcb63b6adecby @localai-org-maint-bot in #12192 - chore: ⬆️ Update ggml-org/llama.cpp to
58367713a6935c0810103378144008df32e3d5dbby @localai-org-maint-bot in #12197 - chore: ⬆️ Update ggml-org/whisper.cpp to
307869af285d7f6f689ba100b3515e2d1b3feb05by @localai-org-maint-bot in #12200 - chore: ⬆️ Update ggml-org/llama.cpp to
709fe755dfa810d77e2ac386292b29648b536864by @localai-org-maint-bot in #12208 - chore: ⬆️ Update ikawrakow/ik_llama.cpp to
c5b5773bed338c5f3b985d277764a4d780b83d42by @localai-org-maint-bot in #12210 - chore: ⬆️ Update PrismML-Eng/llama.cpp to
bdc23b56b4458b9f1655aec5287f3ab56ee8daaaby @localai-org-maint-bot in #12207 - chore: ⬆️ Update ggml-org/whisper.cpp to
a44e07845931421bb6f3447ce0010ed9dc76a118by @localai-org-maint-bot in #12209 - chore: ⬆️ Update 0xShug0/audio.cpp to
1ee4ce8275997a7dcf0e2a5dc3410e509b898d6dby @localai-org-maint-bot in #12211 - chore: ⬆️ Update leejet/stable-diffusion.cpp to
c92d73c408515c94beef32161bb5960764fde7a0by @localai-org-maint-bot in #12212 - chore: ⬆️ Update CrispStrobe/CrispASR to
18d74132d22fa7c967181720310d4ba1df9b7bf0by @localai-org-maint-bot in #12213 - chore: ⬆️ Update mudler/vllm.cpp to
d4738d241271b4d10134a6499f97337f20fcf8ceby @localai-org-maint-bot in #12175 - chore: ⬆️ Update ServeurpersoCom/omnivoice.cpp to
3ac485d0688fc684f5dcf2c95283b745220f9dccby @localai-org-maint-bot in #12191 - chore: ⬆️ Update TheTom/llama-cpp-turboquant to
4deec5587b2963af00bdf80884f3337e02eb7d64by @localai-org-maint-bot in #12154 - chore: ⬆️ Update ikawrakow/ik_llama.cpp to
f3d6e6e3020ddfebad60113845bf521620766da5by @localai-org-maint-bot in #12233 - chore: ⬆️ Update CrispStrobe/CrispASR to
97a35a6e519fda1835f3c8353516384aa8710b8cby @localai-org-maint-bot in #12230 - chore: ⬆️ Update mudler/vllm.cpp to
b24f8094cba9b4f02df71bcff8d41ddc7e88b4efby @localai-org-maint-bot in #12229 - chore: ⬆️ Update 0xShug0/audio.cpp to
9bdd1d908bbd128e9eb405f5a8e38d0defb84c72by @localai-org-maint-bot in #12224 - chore: ⬆️ Update ggml-org/llama.cpp to
d2e54583c7452353eb35d40431281f6ee984332fby @localai-org-maint-bot in #12228 - chore: ⬆️ Update ggml-org/whisper.cpp to
a664346ea5c6dddff3e61a2b7b32dd4514613f50by @localai-org-maint-bot in #12227 - chore: ⬆️ Update PrismML-Eng/llama.cpp to
0324c66521960d67aa7da8687fb1453a79a6565cby @localai-org-maint-bot in #12226 - chore: ⬆️ Update vllm-metal (darwin) to
v0.30.0by @localai-org-maint-bot in #12225 - chore: ⬆️ Update vllm-project/vllm cu130 wheel to
0.30.0by @localai-org-maint-bot in #12214 - chore: ⬆️ Update CrispStrobe/CrispASR to
acc08e3bd3e5c17a3852115f3efa0e1ab30bc47aby @localai-org-maint-bot in #12259 - chore: ⬆️ Update NVIDIA/NeMo-Speech.cpp to
97a15afa5caa9bce5baaa86c1184103877af4101by @localai-org-maint-bot in #12257 - chore: ⬆️ Update leejet/stable-diffusion.cpp to
b167b942f77ecb17e7f78e163a8c32ff7ac95c10by @localai-org-maint-bot in #12255 - chore: ⬆️ Update ggml-org/whisper.cpp to
d09f61a708f3487afa956ff578e60eae5e7a233cby @localai-org-maint-bot in #12253 - chore: ⬆️ Update ikawrakow/ik_llama.cpp to
20f7a72edd7049fe5a87eef2b5e9a50ae109ca4bby @localai-org-maint-bot in #12251 - chore: ⬆️ Update 0xShug0/audio.cpp to
857de2366ed74bdb2c37f85259089e3a0a6b8cb0by @localai-org-maint-bot in #12248 - chore: ⬆️ Update ServeurpersoCom/omnivoice.cpp to
8ab42195a05a9d48a3942b17568c1f3a876e133aby @localai-org-maint-bot in #12249 - chore: ⬆️ Update PrismML-Eng/llama.cpp to
842b1880415d6f508f03b789e5ce70194def7bfdby @localai-org-maint-bot in #12250 - chore: ⬆️ Update ggml-org/llama.cpp to
84e76d8a23162eca70490da131945ebec1f09bf4by @localai-org-maint-bot in #12258 - chore: ⬆️ Update 0xShug0/audio.cpp to
e79205f3e0083d04e812e1a4a376f71be97e9a22by @localai-org-maint-bot in #12269 - chore: ⬆️ Update ikawrakow/ik_llama.cpp to
1aaf7105be6e55a97fa4a9fd6f5bd362b08436dcby @localai-org-maint-bot in #12270 - chore: ⬆️ Update PrismML-Eng/llama.cpp to
adfffbe41b2cabcd51fff326ab045662265062bbby @localai-org-maint-bot in #12271 - chore: ⬆️ Update CrispStrobe/CrispASR to
6b78932d09765406ba0e0154d95bc6289246ceeeby @localai-org-maint-bot in #12273 - chore: ⬆️ Update leejet/stable-diffusion.cpp to
2f886889e6e8b78738d6b87f7191f6018557c551by @localai-org-maint-bot in #12274 - ci: bump Hugo from 0.146.3 to 0.166.0 by @localai-org-maint-bot in #12281
- chore: ⬆️ Update ikawrakow/ik_llama.cpp to
cdf232cc17e410e60c1bc3b85516c4a41199b662by @localai-org-maint-bot in #12288 - chore: ⬆️ Update CrispStrobe/CrispASR to
013ae1624dc40ecf059065d577180722439f804eby @localai-org-maint-bot in #12292 - chore: ⬆️ Update mudler/parakeet.cpp to
2bf88954dc628b32835734e2e9159550a75a1dc6by @localai-org-maint-bot in #12291 - chore: ⬆️ Update 0xShug0/audio.cpp to
94bd4656399180befc141b17bd6696bf84df0a9fby @localai-org-maint-bot in #12289 - chore(deps): bump LocalAGI to 8253de9 (re-dial dropped MCP sessions) by @walcz-de in #12299
- chore: ⬆️ Update ggml-org/llama.cpp to
95887577ab5fead779581a7030a83c7752ff3234by @localai-org-maint-bot in #12272 - chore: ⬆️ Update mudler/vllm.cpp to
c3bebc357385990f721af66a3a6c69328dd4fc6cby @localai-org-maint-bot in #12252 - chore: ⬆️ Update TheTom/llama-cpp-turboquant to
a3d5603d110bda29222d2011596cdc84d7fa532dby @localai-org-maint-bot in #12232 - chore(deps): bump sentence-transformers from 5.7.0 to 6.1.0 in /backend/python/transformers by @dependabot[bot] in #12245
- chore(deps): update numpy requirement from >=2.5.2 to >=2.5.3 in /backend/python/transformers by @dependabot[bot] in #12244
- chore(deps): update transformers requirement from >=5.15.1 to >=5.17.0 in /backend/python/transformers by @dependabot[bot] in #12105
- chore(deps): bump grpcio from 1.83.0 to 1.84.0 in /backend/python/transformers by @dependabot[bot] in #12102
- chore(deps): bump grpcio from 1.83.1 to 1.84.0 in /backend/python/coqui by @dependabot[bot] in #12106
- chore(deps): bump LocalAGI to 7e0947d (no-RAG-DB crash fix, tool filters, per-collection models) by @mudler-agent in #12302
- chore: ⬆️ Update ikawrakow/ik_llama.cpp to
ed27bf7ed25e637692e89cd341d802522a2cee8aby @localai-org-maint-bot in #12313 - chore: ⬆️ Update CrispStrobe/CrispASR to
ec98831d0776ec8a16ccaf93955693eb7ecfbec3by @localai-org-maint-bot in #12314 - chore: ⬆️ Update 0xShug0/audio.cpp to
77491a33c589c53ff18add050095cf35647c8213by @localai-org-maint-bot in #12315 - chore: ⬆️ Update ServeurpersoCom/omnivoice.cpp to
ead199a2bc4c53a57cac90095ae049a111d9e98dby @localai-org-maint-bot in #12316 - chore: ⬆️ Update leejet/stable-diffusion.cpp to
3f8527a46c54ecf4cb4ed6003da8e8982283c73cby @localai-org-maint-bot in #12317 - chore: ⬆️ Update ggml-org/llama.cpp to
4da6337767f973e2b4d0797e5b323d77d8565e4aby @localai-org-maint-bot in #12318 - chore: ⬆️ Update 0xShug0/audio.cpp to
f825d1d1b92af309585aeb656b2a59c44fc603ebby @localai-org-maint-bot in #12343 - chore: ⬆️ Update ikawrakow/ik_llama.cpp to
d741de5074cd424dd3ba7cfc4d9b7649f1eb0463by @localai-org-maint-bot in #12351 - chore: ⬆️ Update ServeurpersoCom/omnivoice.cpp to
53e6c2066150802ad3cd4b655b31c696e78e0019by @localai-org-maint-bot in #12350 - chore: ⬆️ Update ggml-org/whisper.cpp to
6e4ab854f67f743900934a703d5603419384c961by @localai-org-maint-bot in #12349 - chore: ⬆️ Update CrispStrobe/CrispASR to
2cd383a926e3c37334e75eb5d8b8a85bd82ed22dby @localai-org-maint-bot in #12348 - chore: ⬆️ Update localai-org/ced.cpp to
b10237678d1c3b30c77d19f2e63f6c198c7f8d09by @localai-org-maint-bot in #12346 - chore: ⬆️ Update NVIDIA/NeMo-Speech.cpp to
0f706e43cf1fbc031bad1423e05460d3acaeaa1cby @localai-org-maint-bot in #12361 - chore(deps): bump nib to v0.12.1 by @mudler-agent in #12372
- chore: ⬆️ Update localai-org/ced.cpp to
61dec2ab0106f2047ee40062a7075dbf08c523d0by @localai-org-maint-bot in #12365 - chore: ⬆️ Update ikawrakow/ik_llama.cpp to
0821d62a8b356bd1db3c6765551a30bfcc44a6deby @localai-org-maint-bot in #12364 - chore: ⬆️ Update CrispStrobe/CrispASR to
be202c472503a5c7f1d3e568c420865cad02f1c3by @localai-org-maint-bot in #12363 - chore: ⬆️ Update 0xShug0/audio.cpp to
ed96b7307c8daba2ebcf7912af928825f6b14cb9by @localai-org-maint-bot in #12362 - chore: ⬆️ Update mudler/parakeet.cpp to
623a968bccbd2214588df398fcce687cd4218deaby @localai-org-maint-bot in #12347 - chore: ⬆️ Update TheTom/llama-cpp-turboquant to
bcb85fc3ae85efa0f5f392c6c880dfc524923860by @localai-org-maint-bot in #12344 - chore(vllm-cpp): bump to 967883486 (ABI v30), add hf_overrides and Tev1 entries, fix vllm-cpp gallery installs by @mudler-agent in #12379
- chore: ⬆️ Update 0xShug0/audio.cpp to
9a02e61326aaaf9d462b584ca5e0daba22c0abfcby @localai-org-maint-bot in #12389 - chore: ⬆️ Update NVIDIA/NeMo-Speech.cpp to
4c101bc7113f49101a3e11d2c994c519f41939f6by @localai-org-maint-bot in #12388 - chore: ⬆️ Update mudler/parakeet.cpp to
8c8cec0c4564610a0a4b30a8a6f2ead15d1a76fbby @localai-org-maint-bot in #12387 - chore: ⬆️ Update CrispStrobe/CrispASR to
ba8c1ea667f30b1c0e32ef8574cee68d9f30bcf3by @localai-org-maint-bot in #12386 - chore: ⬆️ Update localai-org/voice-detect.cpp to
b74a896f47c6d04fcca0a962ff317528fd0b0019by @localai-org-maint-bot in #12384 - chore: ⬆️ Update ikawrakow/ik_llama.cpp to
32cddbfcefed93896a39c64e7c38c119de8682e6by @localai-org-maint-bot in #12385 - chore: ⬆️ Update ggml-org/llama.cpp to
a4d880fd5c7f88713ded6db9f0111893bd78afa6by @localai-org-maint-bot in #12345 - chore: ⬆️ Update CrispStrobe/CrispASR to
fdc3a0007d68f8d3905f20e9cd4d18b5f193e096by @localai-org-maint-bot in #12413 - chore: ⬆️ Update mudler/parakeet.cpp to
bee7c14dfcc23613df58176c59a40459e7b47095by @localai-org-maint-bot in #12420 - chore: ⬆️ Update ikawrakow/ik_llama.cpp to
d9e286846d6f8232db48ec5c111a4ea3aea675efby @localai-org-maint-bot in #12418 - chore: ⬆️ Update ggml-org/llama.cpp to
a868c3e3c56657f7e8a6231190dbbe90e7dd86c0by @localai-org-maint-bot in #12419
Other Changes
- chore(model-gallery): ⬆️ update checksum by @localai-org-maint-bot in #12015
- chore(model gallery): 🤖 add 1 new models via gallery agent by @localai-org-maint-bot in #12080
- chore(model-gallery): ⬆️ update checksum by @localai-org-maint-bot in #12128
- chore(model-gallery): ⬆️ update checksum by @localai-org-maint-bot in #12156
- chore(model-gallery): ⬆️ update checksum by @localai-org-maint-bot in #12172
- fix(ci): sign backends in the format we verify by @localai-org-maint-bot in #12166
- chore(model-gallery): ⬆️ update checksum by @localai-org-maint-bot in #12194
- chore(model-gallery): ⬆️ update checksum by @localai-org-maint-bot in #12215
- chore(model-gallery): ⬆️ update checksum by @localai-org-maint-bot in #12231
- chore(model-gallery): ⬆️ update checksum by @localai-org-maint-bot in #12256
- chore(model-gallery): ⬆️ update checksum by @localai-org-maint-bot in #12275
- chore(model-gallery): ⬆️ update checksum by @localai-org-maint-bot in #12290
- fix(ci): use Go 1.27 for Darwin backends by @localai-org-maint-bot in #12284
- fix(ci): retain backend digests for release retries by @localai-org-maint-bot in #12160
- chore(website): refresh the counters by @localai-org-maint-bot in #12039
- chore(model gallery): 🤖 add 1 new models via gallery agent by @localai-org-maint-bot in #12223
- chore(model-gallery): propose variant groupings for review by @localai-org-maint-bot in #12180
- chore(model gallery): 🤖 add 1 new models via gallery agent by @localai-org-maint-bot in #12237
- chore(model-gallery): ⬆️ update checksum by @localai-org-maint-bot in #12310
- chore(website): refresh the counters by @localai-org-maint-bot in #12329
- chore(model-gallery): ⬆️ update checksum by @localai-org-maint-bot in #12342
- chore(model-gallery): ⬆️ update checksum by @localai-org-maint-bot in #12366
- test(agentpool): pin the standalone agent contract by @mudler-agent in #12378
- chore(model-gallery): ⬆️ update checksum by @localai-org-maint-bot in #12390
- chore(model-gallery): ⬆️ update checksum by @localai-org-maint-bot in #12415
New Contributors
- @mudler-agent made their first contribution in #12124
- @lqp made their first contribution in #12031
- @H-XX-D made their first contribution in #12135
- @yzxcj797 made their first contribution in #11546
- @PINYOPATTANAWASANPORN made their first contribution in #12216
- @pratikgx made their first contribution in #12320
Full Changelog: v4.10.0...v4.11.0
AI summary LocalAI adds fleet management, private model access, and backend fixes
🎉 LocalAI 4.10.0 Release! 🚀
LocalAI 4.10.0 is out!
Twenty-eight days and 280 pull requests. The work landed on three fronts: operating a fleet, feeding it models from private sources, and fixing the backends you depend on. A fleet operations dashboard replaces the flat node list with cluster-wide health, capacity, running models, and bulk lifecycle actions. A credentials file lets one credentials.yaml authenticate OCI registries, galleries and direct downloads without scattering tokens across environment variables. And local-ai benchmark measures end-to-end latency and throughput from the CLI.
The ds4 backend had four fixes that together make it reliable on CUDA: the kernels now build for the target architecture, abandoned inference is cancelled, generation boundaries are enforced, and prefilled reasoning is separated from content. Distributed mode gained seven stabilisation fixes across staging, liveness, virtual models and the control-plane database. Four CVEs were patched, enable_thinking=false is now honoured across every text backend, and Apple M5 no longer SIGSEGVs at startup.
Highlights:
- 📊 Fleet operations dashboard - a real-time cluster overview replacing the flat node list. Aggregate health bands, capacity gauges for VRAM/RAM/CPU/disk, a running-models tab showing replica placement across workers, bulk drain/resume/remove with bounded concurrency, and a slide-out node inspector. Workers now report CPU telemetry at registration and heartbeat.
- 🔑 Credentials file for private sources - one
credentials.yamlauthenticates OCI registries, galleries, HTTP downloads andgithub:URIs. Supports basic auth, bearer tokens and custom headers, with secrets read at use time so rotated K8s secret mounts take effect without restart, and full masking in logs and error messages. - 📏
local-ai benchmarkCLI command - measures end-to-end latency and throughput against configured text models. Reports min/mean/max latency and completion tokens/sec across warmup and measured runs, with table and JSON output, redirect blocking, and API-key redaction. - 🔧 vllm-cpp LoRA fusion at load time -
lora_adaptersandlora_scalesare now consumed by the vllm-cpp video engine, fusing LoRA deltas into DiT weights at engine load. Works for both LTX2.5 and MiniMax-H3 through one generic path. - 🌍 Environment variables per model - an
env:map in model config injects variables into the backend process environment, soCUDA_VISIBLE_DEVICESorVLLM_CACHE_DIRcan be pinned per model without wrapping the backend in a script. - 💬
template.system_messages_after_first- agent frameworks that append system instructions mid-conversation no longer crash Qwen3.x models. Late system turns can be merged into the leading system message or forwarded as user turns. - 🧠
context_sizein/v1/models/capabilities- clients can now read each model's effective context window to budget tokens and truncate prompts without probing the model config out-of-band. - 🛡️ Four CVEs patched - ip-address 10.3.1 (CVE-2026-69192), containerd 1.7.32 and 1.7.33 (CVE-2026-46680, CVE-2026-53488), and react-router (GHSA-qwww-vcr4-c8h2).
- 🍎 Apple M5 SIGSEGV fixed -
go-m1cpubumped to v0.2.2, fixing a segmentation fault on Apple M5 hardware at startup.
Plus a focus mode toggle in chat settings, face enrollment replay for the 1:N registry, a ROCm build of stablediffusion-ggml, in-place editing of distributed scheduling rules, OCI layer download resumption, and 140 new gallery entries.
📊 This release in numbers
| Pull requests merged | 280 |
| Commits | 382 |
| Files changed | 633 (+46,550 / -3,316) |
| Development window | 28 days (2026-08-20 to 2026-09-17) |
| Human contributors | 29, of whom 16 first-time |
| Gallery entries | 1,707 to 1,847 (+140) |
Where the work landed:
| Area | Change |
|---|---|
core/ |
+25,773 / -1,993 across 314 files |
gallery/ |
+6,734 / -417 |
backend/ |
+4,932 / -433 across 154 files |
pkg/ |
+4,392 / -91 across 55 files |
docs/ |
+2,526 / -101 across 48 files |
swagger/ |
+322 |
📌 TL;DR
| Area | Summary |
|---|---|
| 📊 Fleet dashboard | The Nodes page is rewritten as a fleet operations dashboard. A ClusterOverview shows health bands (healthy/draining/attention) and capacity gauges for VRAM, RAM, CPU and disk derived from the single GET /api/nodes response. A NodeFleetTable offers sortable, groupable, paginated rows with checkbox selection across filters and pages, and bulk Drain/Resume/Remove with bounded concurrency (limit 8). A Running Models tab shows all loaded replicas on healthy workers grouped by model, with replica count, node count, in-flight requests and backend types. Slide-out inspectors for nodes and models provide detail and drill-down. Workers now report cpu_logical_cores, cpu_usage_percent and cpu_load_1 at registration and heartbeat. Lifecycle transitions are atomic: MarkDraining flips status and deletes model records in one transaction, ResumeNode is conditional (draining to healthy only), and HTTP endpoints return proper 404/409 instead of generic 500. |
| 🔑 Credentials file | --credentials-file / LOCALAI_CREDENTIALS_FILE loads a YAML list of rules, each matching a URL prefix and carrying one auth type: basic (username + password/password_env/password_file), bearer (bearer/bearer_env/bearer_file), or custom header (header.name + header.value/_env/_file). OCI registries consult the store first, then docker config. HTTP downloads, galleries and github: URIs authenticate through a RoundTripper that clones the request per redirect hop, so a cross-host redirect never carries the credential. Secrets are read at use time, not load time, so a rotated K8s secret mount takes effect without restart. Credential and Store implement String, GoString and LogValue so no fmt verb or slog call can print secret material. Unknown YAML keys are errors; match strings with user:token@, ? or # are rejected at load. |
| 📏 CLI benchmark | local-ai benchmark MODEL [MODEL ...] sends non-streaming POST /v1/chat/completions requests at temperature:0 with configurable --runs (default 3), --warmup (default 1), --max-tokens (default 128) and --timeout (default 5m). Reports min/mean/max latency and end-to-end completion tokens/sec per model. Output is a table by default or --json for structured data with per-sample latency and nullable token counts. The HTTP client refuses redirects, the API key is redacted from error messages, and the full report is buffered and only written after all models succeed, so a failure leaves no partial output. |
| 🔧 vllm-cpp LoRA fusion | The lora_adapters, lora_scales, lora_adapter and lora_scale config fields are now consumed by the vllm-cpp video backend. buildLoraExtras converts them into indexed lora_path/lora_strength key-value pairs (first adapter unsuffixed, subsequent ones _2, _3, ...), packs them via the ABI v18 extras seam, and hands them to vllmVideoEngineLoad. vllm.cpp fuses the LoRA deltas into the DiT weights at load. One generic path serves both LTX2.5 and MiniMax-H3. Adapters are always loaded; per-request prompt-activated LoRA is out of scope. |
| 🌍 Per-model env vars | An env: map on the model config is appended to the child process environment after the base env, so model vars override inherited system vars. Delivered through the common process-spawning layer in pkg/model/process.go, so every gRPC backend inherits it with no per-backend wiring. The UI renders an editable key/value map. Model vars take effect only when a model spawns its own backend process, not when it reuses a worker-supervised one. |
| 💬 Late system messages | template.system_messages_after_first controls what happens to system-role messages after the leading system block. "merge" folds them into the first system message with a \n\n separator; "user" rewrites them in place to role user; unset passes through. Fixes a Jinja exception (System message must be at the beginning) raised by Qwen3.x tokenizer templates when agent frameworks append system instructions mid-conversation. Runs after stripEmptySystemMessages and before tokenizer-template system_prompt injection. |
| 🧠 context_size exposed | /v1/models/capabilities now populates entry.ContextSize with backend.EffectiveContextSize(cfg) when the result is positive, letting clients budget tokens and truncate prompts without probing the model config out-of-band. |
| 🛡️ CVE fixes | ip-address to 10.3.1 (CVE-2026-69192), containerd to 1.7.32 (CVE-2026-46680) then 1.7.33 (CVE-2026-53488), react-router to 7.18.2 / 8.3.0 (GHSA-qwww-vcr4-c8h2). |
| 🍎 Apple M5 | go-m1cpu bumped to v0.2.2, fixing a SIGSEGV on Apple M5 at startup. |
| 🧠 Models | 140 new gallery entries: Qwopus 3.6, OpenResearcher, LFM2.5 DSpark, Qwen3.8 small distills and Flash Next, UI-Mate 27B, Homura 30B, Ornith 1.5 35B and 9B, Granite 4.2, Tiel-Coder 35B, Thomson 1.0 Small, PhoneLLM, WeMM embedding, EXL3 vllm.cpp, Gemma 4 E4B, Orukeet for NeMo speech, and more. |
🚀 New Features & Major Enhancements
📊 Fleet operations dashboard
The Nodes page was a flat list of per-node cards. It showed you what was registered but not what the cluster was doing, and every lifecycle action was one node at a time.
The rewrite is a fleet operations dashboard with two tabs and a shared summary poller.
ClusterOverview sits at the top. A health band segments nodes into healthy, draining and attention counts. Capacity gauges show VRAM, RAM, CPU and disk, each reporting how many workers do not report that metric so the number is honest rather than averaged over reporting nodes. An attention queue filters for pending approval, offline/unhealthy, low VRAM, low RAM and low disk, surfacing nodes that need action without double-counting the headline total.
NodeFleetTable is sortable, groupable (by label or type) and paginated at 50 rows per page, with checkbox selection that persists across filters and pages. Bulk Drain, Resume and Remove run with bounded concurrency (limit 8) so a fleet with thousands of registrations is usable. Columns: status pill, address, capacity (VRAM + RAM), CPU, workload and heartbeat. An Approve button appears on pending nodes.
Running models tab. A fleet-wide view of loaded replicas on healthy workers, grouped by model: model name, replica count, node count, in-flight requests, backend types and last used. Per-model Stop shuts down all placements through POST /backend/shutdown {model}, which now fires the remote unloader even when a local process exists, and deduplicates nodes before sending stop messages.
Inspectors. A slide-out NodeInspector shows health, labels, resource bars, CPU, loaded model count, backends (lazy-loaded) and in-flight, with Approve/Drain/Resume actions. A ModelInspector shows replica placement grouped by node, with clickable node names that drill into a NodeInspector.
Workers now report cpu_logical_cores, cpu_usage_percent and cpu_load_1 at registration and heartbeat. The dashboard shows CPU capacity and utilization, with idle cores derived as cpu_logical_cores * (1 - cpu_usage_percent / 100).
Lifecycle transitions are now atomic. MarkDraining flips status and deletes model records in one transaction rather than as separate operations that could interleave with SetNodeModel. ResumeNode is conditional: draining to healthy only, preventing overwriting of pending approval or concurrent state changes. HTTP endpoints return proper 404 (not found) and 409 (status conflict) instead of a generic 500.
🔗 PRs: #12046
🔑 Credentials file for private sources
Backends, models and gallery indexes can live behind private OCI registries, internal HTTP servers or private GitHub repos. Before this, each download path had its own ad-hoc auth mechanism, and tokens were scattered across environment variables.
One credentials.yaml file, loaded by --credentials-file / LOCALAI_CREDENTIALS_FILE, authenticates them all. It is a top-level YAML list of rules. Each rule has a match URL prefix and exactly one auth type: basic (username + password / password_env / password_file), bearer (bearer / bearer_env / bearer_file), or custom header (header.name + header.value / header.value_env / header.value_file). Exactly one of the plain, _env or _file form per secret; unknown keys are errors so a misspelled pasword_env fails loudly.
OCI registries (backend images, oci:// and ollama:// models, digest lookup, Range resume, cosign verify) consult the store first, then authn.DefaultKeychain (docker config). HTTP downloads, galleries and github: URIs authenticate through an http.RoundTripper that puts the credential on a clone of the request. net/http rebuilds each redirect hop from the original headers, so a redirect to another host never carries the credential. A caller-supplied credential (e.g. HF_TOKEN) takes precedence: the transport leaves an existing Authorization header alone.
Secrets are read at use time, not load time, so a rotated K8s secret mount takes effect without restart. Credential and Store implement String, GoString and LogValue, so no fmt verb or slog call, including when nested in unexported fields, can print secret material. Credentials go only to https, or to http when the rule sets allow_insecure. Match strings containing user:token@, ? (where signed URLs carry tokens) or # are rejected at load. YAML decode errors are redacted: quoted scalar values (often secrets) are stripped, keeping only line numbers and parser wording.
🔗 PRs: #12028
📏 local-ai benchmark CLI command
A benchmark command for configured text models, answering "how fast is this model on this hardware" without leaving the terminal.
local-ai benchmark MODEL [MODEL ...] sends non-streaming POST /v1/chat/completions requests at temperature:0 with configurable --runs (default 3), --warmup (default 1), --max-tokens (default 128) and --timeout (default 5m). It reports min, mean and max latency and end-to-end completion tokens/sec per model.
The throughput metric is sum(completion_tokens) / sum(latencies) across measured runs. It is nil/N/A if any sample omits completion_tokens from the response usage, or if total latency is zero. The command does not measure decode-only speed or time to first token; it is an end-to-end client benchmark covering HTTP transport, queueing, prompt processing, generation and parse.
The HTTP client refuses redirects by default, so a misconfigured endpoint behind a redirect does not silently succeed. The API key is redacted from error messages, and raw transport errors are masked as HTTP request failed to avoid credential echo. The full report is buffered in a bytes.Buffer and only written via io.Copy after all models succeed, so a failure leaves no partial output. Signal handling cancels the active request on SIGINT/SIGTERM.
Output is a table by default or --json for structured data with per-sample latency and nullable token counts. The API key is not included in the JSON. To compare backends, configure separate model aliases per backend and pass those alias names.
🔗 PRs: #12020
🔧 vllm-cpp LoRA fusion at load time
The lora_adapters and lora_scales config fields are standard LocalAI fields that list LoRA adapter weight files and per-adapter strength multipliers. The vllm-cpp video backend now consumes them for the first time, fusing LoRA deltas into DiT (diffusion Transformer) weights at engine load.
buildLoraExtras converts the config fields into indexed lora_path / lora_strength key-value pairs. The first adapter is unsuffixed; subsequent ones get _2, _3, etc. If the singular lora_adapter is set, it is prepended as the first adapter before the plural list. Relative adapter paths are joined to the model path; absolute paths pass through. Strength defaults to 1.0 when no scale is provided.
The pairs are packed via the ABI v18 extras seam and handed to vllmVideoEngineLoad. vllm.cpp's ResolveDitLoraSpecs fuses the LoRA deltas into the DiT weights at load. One generic path serves both LTX2.5 and MiniMax-H3, since the engine auto-detects the DiT family from the checkpoint.
Adapters are always loaded; there is no per-request activation in this path. Runtime prompt-activated LoRA (punica-style <lora:name:strength>) is out of scope and tracked separately.
🔗 PRs: #12082
🌍 Environment variables per model
An env: map on the model config injects variables into the backend process environment. The primary use case is pinning CUDA_VISIBLE_DEVICES per model to control GPU placement, but it applies to any backend that reads environment variables.
name: vllm-model backend: vllm env: CUDA_VISIBLE_DEVICES: "0,1" VLLM_CACHE_DIR: "/tmp/vllm_cache"
The map is delivered through the common process-spawning layer in pkg/model/process.go, so every gRPC backend inherits it with no per-backend wiring. Model vars are appended last, after the base env, so they override same-named inherited system vars. The UI renders an editable key/value map.
Model vars take effect only when a model spawns its own backend process, not when it reuses a worker-supervised one. There is no allowlist or blocklist: model configs are an operator trust boundary, consistent with LocalAI's general posture.
🔗 PRs: #10721
💬 template.system_messages_after_first
Agent frameworks legitimately append system instructions mid-conversation: a tool-parameter prompt before a call, an adjustment prompt after a result. The Qwen3.x family of tokenizer templates raises Jinja Exception: System message must be at the beginning for any system-role message after the first turn, returning HTTP 500. Plain chat looks healthy; every agent task fails.
template.system_messages_after_first controls what happens to system-role messages that appear after the contiguous leading system block:
"merge"folds late system messages into the first system message with a\n\nseparator, dropping them from their original position."user"rewrites each late system message in place to roleuser, keeping its position and content.- unset (default) passes through unchanged; the tokenizer template decides.
The normalization runs in ChatEndpoint after stripEmptySystemMessages and before tokenizer-template system_prompt injection, so the Jinja template never sees an out-of-place system message. It only touches requests whose model config opts in, so default behavior is unchanged.
"user" is the recommended mode for agent workloads: the instruction stays where the framework placed it, and the model follows it.
🔗 PRs: #11906
🧠 context_size in /v1/models/capabilities
The capabilities endpoint now populates entry.ContextSize with backend.EffectiveContextSize(cfg) when the result is positive. Clients can read each model's effective context window to budget tokens, truncate prompts and history, and avoid sending more context than the model accepts, without having to probe the model config out-of-band.
🔗 PRs: #12083
🕵️ Face enrollment replay
The face recognition 1:N registry gains a replay path. FaceRegisterRequest now accepts an Embedding (pre-computed vector) and RegisteredAt (original timestamp), so a client that previously extracted an embedding via /v1/face/embed can replay the saved vector, name, labels and timestamp directly, skipping image decoding and the FaceEmbed inference call entirely.
The store derives a deterministic ID from the embedding bytes and store namespace via uuid.NewSHA1, so replaying the same vector after a restart or on another replica returns the same ID without creating duplicates. The server does not persist the registry itself; clients must retain and replay the records to restore the registry after a restart.
🔗 PRs: #11908
🎯 Focus mode toggle
Focus mode (which collapses the global app sidebar and slims the header once a conversation has at least one message, with Esc temporarily restoring them) already existed as session-only behavior. This adds a persistent opt-out toggle in the chat settings drawer, backed by localStorage under localai_chat_focus_mode. It defaults to enabled; with the toggle off, the full layout stays even in an active conversation.
🔗 PRs: #11750
🧰 Smaller features worth knowing about
stablediffusion-ggmlbuilds a ROCm variant. A ROCm build of the stable-diffusion ggml backend, extending accelerator coverage beyond CUDA and Metal.- Scheduling rules editable in place. Distributed scheduling rules can now be edited in place in the UI rather than through a round-trip to the config file.
- Scheduling rules keyed by model alias. Distributed scheduling rules can be keyed by a model alias, not just a model name, so aliased models participate in scheduling correctly.
- XIO copy buffer size is configurable. The copy buffer size for XIO transfers is now configurable rather than fixed.
- OCI layer download resumption. Interrupted OCI layer downloads resume rather than restarting from scratch.
- Progress updates no longer throttle downloads. Status updates were throttling downloads; the progress path is now bounded so it does not compete with the transfer.
- Node operation controls restored. The node detail page's operation controls (drain, resume, remove) are restored after a regression that hid them, and backend actions are tested through their menu.
🔗 PRs: #11774, #11667, #11771, #11660, #11688, #11661, #12068, #12069
🐛 Bug Fixes (recap)
Security / CVE:
fix: upgrade ip-address to 10.3.1 (CVE-2026-69192) - #11632fix: upgrade containerd to 1.7.32 (CVE-2026-46680) - #11634fix: upgrade react-router to 7.18.2, 8.3.0 (GHSA-qwww-vcr4-c8h2) - #11644fix: upgrade containerd to 1.7.33 (CVE-2026-53488) - #11655fix(auth): bypass API-key auth for CORS preflight (OPTIONS) requests - #11113
ds4:
fix(ds4): separate prefilled reasoning from content - #11802fix(ds4): enforce generation boundaries - #11821fix(ds4): cancel abandoned inference - #11822fix(ds4): build CUDA kernels for the target architecture - #11840fix(ds4): resolve Hugging Face repository imports - #11340
Distributed mode:
fix(distributed): restore node liveness tests - #11694fix(distributed): survive a slow control-plane database - #11837fix(distributed): stage sound detection audio - #11907fix(distributed): finalize stalled model uploads - #11910fix(distributed): resolve paths for virtual models - #11911fix(distributed): bound ephemeral staging - #11924fix(distributed): pass raw JPEG base64 through staging - #11956fix(distributed): keep staging operations stable - #11663fix(distributed): keep model replicas config-consistent - #11664
Realtime:
fix(realtime): accept GA WebRTC signaling - #11778fix(realtime): skip responses for empty transcripts - #11940fix(realtime): resolve pipeline voice profiles - #11942fix(realtime): support voice profile switching - #11948
Backends:
fix(backends): honor enable_thinking=false in sglang and vllm - #11715fix(backends): honor enable_thinking=false in mlx and vllm-omni - #11962fix(diffusers): auto-detect CUDA instead of defaulting to CPU - #11891fix: prefer the Intel diffusers venv SYCL runtime at startup - #11971fix(qwen-tts): restore CUDA 13 SDPA fallback - #12067fix(faster-whisper): manually install ctranslate2 with rocm support - #9839fix(whisper): honour positional listen address argument - #11652fix(backends): bound temporary scratch files - #11941fix(openresponses): support Codex WebSocket warm-up - #11608
Gallery and models:
fix(gallery): tag MTP-enabled models - #11653fix(gallery): remove duplicate Qwopus entries - #11670fix(gallery): use published F16 mmproj for qwythos-9b - #11792fix(gallery): persist inference defaults where the loader reads them - #11232fix(gallery): default audio-cpp models to backend:best - #11892fix(gallery): restore entries clobbered by batch PR applies - #12016fix(gallery): use native Ministral tool parsing - #12022fix(vram): persist remote probe metadata - #11487fix(ollama): accept :latest tag on model lookup - #11732
UI:
fix(ui): omit empty system prompt so model default applies - #11838fix(ui): omit GPU recommendations that do not fit - #11945fix(ui): preserve percent signs in route parameters - #11883fix(ui): restore node operation controls - #12068fix(ui): test backend actions through their menu - #12069fix(ui): size model fit against the cluster - #11765
Other:
fix(deps): bump go-m1cpu to v0.2.2 to fix SIGSEGV on Apple M5 - #11736fix(downloader): make file:// installs reachable again - #11734fix(detection): avoid temporary image files - #11938fix(worker): resolve temporary paths in tests - #11944fix(ci): bound Discord release summaries - #11695fix(ci): stop the e2e teardown from failing a green suite - #11816fix(ci): unbreak the e2e build and the darwin vllm-metal pin - #11849fix(progress): stop status updates throttling downloads - #11661fix(oci): resume interrupted layer downloads - #11688
🧠 Models
140 new gallery entries this cycle, taking the index from 1,707 to 1,847.
Text generation: Qwopus 3.6, OpenResearcher, LFM2.5 DSpark, Qwen3.8 small distills, Qwen3.8 Flash Next, Qwen3.8 OBLITERATED, Qwen3.8 Cold Fusion, Qwen3.8 GSQ-RCO, UI-Mate 27B, Homura 30B, Ornith 1.5 35B and 9B (including OBLITERATED), Granite 4.2, Tiel-Coder 35B, Thomson 1.0 Small, PhoneLLM, and Huihui Qwen3.8 Flash Next.
Embeddings: WeMM embedding variants.
EXL3: vllm.cpp EXL3 models.
Vision: Gemma 4 E4B HauhauCS variants.
Speech: Orukeet added to the NeMo speech backend.
🔗 PRs: #11662, #11658, #11676, #11675, #11672, #11680, #11691, #11714, #11719, #11723, #11766, #11772, #11763, #11775, #11779, #11754, #11751, #11787, #11803, #11912, #11625, #11903, #11921, #11998, #11716
👒 Dependencies
Submodule and pin bumps this cycle:
| Project | Bumps |
|---|---|
| 0xShug0/audio.cpp | 16 |
| ggml-org/llama.cpp | 12 |
| CrispStrobe/CrispASR | 10 |
| ikawrakow/ik_llama.cpp | 8 |
| leejet/stable-diffusion.cpp | 7 |
| mudler/vllm.cpp | 6 |
| ggml-org/whisper.cpp | 6 |
| antirez/ds4 | 6 |
| ServeurpersoCom/omnivoice.cpp | 5 |
| NVIDIA/NeMo-Speech.cpp | 4 |
| PrismML-Eng/llama.cpp | 3 |
| mudler/depth-anything.cpp | 2 |
| vllm-project/vllm cu130 wheel, vllm-metal (darwin), OpenNMT/CTranslate2 ROCm wheel, TheTom/llama-cpp-turboquant | 1 each |
Plus vllm 0.26.0 to 0.29.0 in the Python backends, torch 2.13.0+xpu to 2.14.0+xpu, grpcio to 1.83.1 across coqui/rerankers/vllm/template, charset-normalizer to 3.5.1, securego/gosec to 2.29.0, LocalAGI and localrecall to v0.6.5, 16 stale dependabot entries removed, npm_and_yarn group updates, and inference defaults refreshed from unsloth.
📖 Documentation
New pages for the CLI benchmark command and the credentials file / private sources guide. The distributed-mode docs document the fleet dashboard, CPU telemetry and the new endpoint error codes. Model configuration gained the env: map, template.system_messages_after_first and context_size documentation. The DCO guide clarifies that maintainer-operated automation may sign off. Installation paths were simplified, an Apache APISIX reverse proxy example was added, dead anchors and section links were fixed, documented env var and CLI flag names were corrected, and the containers page was fixed for Podman image qualification. A FAQ note covers OPENAI_BASE_URL for pointing clients at LocalAI, and Solstone was added to the README and ADOPTERS.md.
🔗 PRs: #11631, #11294, #11850, #11700, #11885, #11884, #11886, #11749, #11904, #11310, #12044, #12045
🙌 New Contributors
- @localai-org-bot made their first contribution in #11631
- @DanielSwift1992 made their first contribution in #11686
- @leilei3167 made their first contribution in #11715
- @sz-po made their first contribution in #11736
- @blackd made their first contribution in #11750
- @ginavalent made their first contribution in #11806
- @FiloSpaTeam made their first contribution in #11802
- @aimansour made their first contribution in #11700
- @simpleqt made their first contribution in #11885
- @alex-mazzariol made their first contribution in #11749
- @seven7763 made their first contribution in #11310
- @QiuLsG made their first contribution in #11883
- @SomSamantray made their first contribution in #11652
- @devv-shayan made their first contribution in #11962
- @Nathan-Roll1 made their first contribution in #11998
- @quartzjer made their first contribution in #12045
Thanks also to @mudler, @anupamme, @mvanhorn, @jimmykarily, @Nold360, @Anai-Guo, @SuperMarioYL, @eglia and @walcz-de for their contributions this cycle.
What's Changed
Bug fixes 🐛
- fix: upgrade ip-address to 10.3.1 (CVE-2026-69192) by @anupamme in #11632
- fix: upgrade github.com/containerd/containerd to 1.7.32 (CVE-2026-46680) by @anupamme in #11634
- fix: upgrade react-router to 7.18.2, 8.3.0 (GHSA-qwww-vcr4-c8h2) by @anupamme in #11644
- fix(gallery): tag MTP-enabled models by @localai-org-maint-bot in #11653
- fix: upgrade github.com/containerd/containerd to 1.7.33 (CVE-2026-53488) by @anupamme in #11655
- fix(gallery): remove duplicate Qwopus entries by @localai-org-maint-bot in #11670
- fix(vram): persist remote probe metadata by @localai-org-maint-bot in #11487
- fix(distributed): restore node liveness tests by @localai-org-maint-bot in #11694
- fix(backends): honor enable_thinking=false in sglang and vllm by @leilei3167 in #11715
- fix(ollama): accept :latest tag on model lookup by @leilei3167 in #11732
- fix(deps): bump go-m1cpu to v0.2.2 to fix SIGSEGV on Apple M5 by @sz-po in #11736
- fix(realtime): accept GA WebRTC signaling by @localai-org-maint-bot in #11778
- fix(gallery): use published F16 mmproj for qwythos-9b by @leilei3167 in #11792
- fix(ds4): separate prefilled reasoning from content by @FiloSpaTeam in #11802
- fix(ds4): enforce generation boundaries by @FiloSpaTeam in #11821
- fix(distributed): survive a slow control-plane database by @localai-org-maint-bot in #11837
- fix(ui): omit empty system prompt so model default applies by @leilei3167 in #11838
- fix(ds4): cancel abandoned inference by @FiloSpaTeam in #11822
- fix(ds4): build CUDA kernels for the target architecture by @FiloSpaTeam in #11840
- fix(gallery): persist inference defaults where the loader reads them by @jimmykarily in #11232
- fix(downloader): make file:// installs reachable again (#11701) by @Anai-Guo in #11734
- fix(distributed): stage sound detection audio by @localai-org-maint-bot in #11907
- fix(distributed): finalize stalled model uploads by @localai-org-maint-bot in #11910
- fix(distributed): resolve paths for virtual models by @localai-org-maint-bot in #11911
- fix(diffusers): auto-detect CUDA instead of defaulting to CPU by @blackd in #11891
- fix(gallery): default audio-cpp models to backend:best by @blackd in #11892
- fix(distributed): bound ephemeral staging by @localai-org-maint-bot in #11924
- fix(detection): avoid temporary image files by @localai-org-maint-bot in #11938
- fix(realtime): skip responses for empty transcripts by @localai-org-maint-bot in #11940
- fix(realtime): resolve pipeline voice profiles by @localai-org-maint-bot in #11942
- fix(backends): bound temporary scratch files by @localai-org-maint-bot in #11941
- fix(worker): resolve temporary paths in tests by @localai-org-maint-bot in #11944
- fix(realtime): support voice profile switching by @localai-org-maint-bot in #11948
- fix(distributed): pass raw JPEG base64 through staging by @localai-org-maint-bot in #11956
- fix(ui): omit GPU recommendations that do not fit by @localai-org-maint-bot in #11945
- fix: prefer the Intel diffusers venv SYCL runtime at startup by @mvanhorn in #11971
- fix(ds4): resolve Hugging Face repository imports by @localai-org-maint-bot in #11340
- fix(openresponses): support Codex WebSocket warm-up by @aimansour in #11608
- fix(ui): preserve percent signs in route parameters by @QiuLsG in #11883
- fix(whisper): honour positional listen address argument by @SomSamantray in #11652
- fix(auth): bypass API-key auth for CORS preflight (OPTIONS) requests by @SuperMarioYL in #11113
- fix(backends): honor enable_thinking=false in mlx and vllm-omni by @devv-shayan in #11962
- fix(faster-whisper): manually install ctranslate2 with rocm support before installing other dependencies by @eglia in #9839
- fix(gallery): restore entries clobbered by batch PR applies by @localai-org-maint-bot in #12016
- fix(gallery): use native Ministral tool parsing by @localai-org-maint-bot in #12022
- fix(ui): restore node operation controls by @localai-org-maint-bot in #12068
- fix(qwen-tts): restore CUDA 13 SDPA fallback by @localai-org-maint-bot in #12067
- fix(ui): test backend actions through their menu by @localai-org-maint-bot in #12069
Exciting New Features 🎉
- feat(ui): add 'Focus mode' option in chat settings to persistently toggle the sidebar auto-collapse by @blackd in #11750
- feat(faces): replay saved face enrollments by @localai-org-maint-bot in #11908
- [chat] feat: template.system_messages_after_first — merge or forward late system turns by @walcz-de in #11906
- [model-config] feat: add environment variables support for backends by @Nold360 in #10721
- feat(cli): benchmark configured text models by @localai-org-maint-bot in #12020
- feat(downloads): authenticate registries, galleries and downloads from a credentials file by @localai-org-maint-bot in #12028
- feat(nodes): add fleet operations dashboard by @localai-org-maint-bot in #12046
- feat: expose context_size in /v1/models/capabilities by @localai-org-maint-bot in #12083
- [vllm-cpp] wire lora_adapters/lora_scales into DiT load-time fusion by @localai-org-maint-bot in #12082
🧠 Models
- feat(gallery): add Qwopus 3.6 variants by @localai-org-maint-bot in #11662
- feat(gallery): add OpenResearcher variants by @localai-org-maint-bot in #11658
- feat(gallery): add LFM2.5 DSpark variants by @localai-org-maint-bot in #11676
- feat(gallery): add Qwen3.8 small distills by @localai-org-maint-bot in #11675
- feat(gallery): add UI-Mate 27B variants by @localai-org-maint-bot in #11672
- feat(gallery): add Homura 30B by @localai-org-maint-bot in #11680
- feat(gallery): add Qwen3.8 OBLITERATED variants by @localai-org-maint-bot in #11691
- feat(gallery): add Ornith 1.5 35B variants by @localai-org-maint-bot in #11714
- feat(gallery): add Granite 4.2 variants by @localai-org-maint-bot in #11719
- feat(gallery): add Tiel-Coder 35B variants by @localai-org-maint-bot in #11723
- feat(gallery): add Thomson 1.0 Small variants by @localai-org-maint-bot in #11766
- feat(gallery): add PhoneLLM variants by @localai-org-maint-bot in #11772
- feat(gallery): add Qwen3.8 Flash Next by @localai-org-maint-bot in #11763
- feat(gallery): add WeMM embedding variants by @localai-org-maint-bot in #11775
- feat(gallery): add Granite 4.2 variants by @localai-org-maint-bot in #11779
- feat(gallery): add Qwen3.8 Cold Fusion by @localai-org-maint-bot in #11754
- feat(gallery): add LFM2.5 8B DSpark variants by @localai-org-maint-bot in #11751
- feat(gallery): add Qwen3.8 GSQ-RCO variants by @localai-org-maint-bot in #11787
- feat(gallery): add Ornith 1.5 9B OBLITERATED by @localai-org-maint-bot in #11803
- feat(gallery): add EXL3 vllm.cpp models by @localai-org-maint-bot in #11912
- feat(gallery): add Ornith 1.5 35B variants by @localai-org-maint-bot in #11625
- feat(gallery): add Gemma 4 E4B HauhauCS variants by @localai-org-maint-bot in #11903
- feat(gallery): add Huihui Qwen3.8 Flash Next by @localai-org-maint-bot in #11921
- [gallery] feat: add Orukeet to the NeMo speech backend by @Nathan-Roll1 in #11998
📖 Documentation and examples
- docs(website): simplify installation paths by @localai-org-bot in #11631
- docs: add Apache APISIX reverse proxy example by @localai-org-maint-bot in #11294
- docs(dco): let maintainer-operated automation sign off by @localai-org-maint-bot in #11850
- docs(llama-cpp): clarify multimodal speculative decoding by @aimansour in #11700
- docs: fix dead anchors and a dead section link by @simpleqt in #11885
- docs: fix dead links in the PR template and backends docs by @simpleqt in #11884
- docs: correct documented env var and CLI flag names by @simpleqt in #11886
- Update containers.md to fix podman image qualification by @alex-mazzariol in #11749
- docs(integrations): add Distribution Packages section by @blackd in #11904
- docs(faq): note OPENAI_BASE_URL env var for pointing clients at LocalAI by @seven7763 in #11310
- Add Solstone app link and image to README by @mudler in #12044
- Add solstone to ADOPTERS.md by @quartzjer in #12045
👒 Dependencies
- chore(deps): update charset-normalizer requirement from >=3.4.9 to >=3.5.1 in /backend/python/vllm by @dependabot[bot] in #11637
- chore(deps): remove 16 dependabot entries for directories that no longer exist by @DanielSwift1992 in #11686
- chore(deps): bump vllm from 0.26.0 to 0.28.0 in /backend/python/vllm by @dependabot[bot] in #11752
- chore: ⬆️ Update leejet/stable-diffusion.cpp to
6b3edaaf32cc19e5bb2d819c788bd557eddc8ebaby @localai-org-maint-bot in #11793 - chore: ⬆️ Update ggml-org/whisper.cpp to
eacbd8234c6654cdbf2c377f72b2106875479bdcby @localai-org-maint-bot in #11796 - chore: ⬆️ Update CrispStrobe/CrispASR to
18b3e3f8456748a6380dc4c13817df244b695d39by @localai-org-maint-bot in #11799 - chore: ⬆️ Update ikawrakow/ik_llama.cpp to
3c58ae373a0081c884099f435fb16ca720852bf7by @localai-org-maint-bot in #11809 - chore: ⬆️ Update mudler/vllm.cpp to
6a544bdb89eb5a3512ac922241439e45f24d74d4by @localai-org-maint-bot in #11797 - chore: ⬆️ Update 0xShug0/audio.cpp to
bf3315fe4aaa16dc1125f580c29aff90a8900b36by @localai-org-maint-bot in #11794 - chore: ⬆️ Update mudler/depth-anything.cpp to
02ba082274e001a63e50de5a1eb0ccc50c6af4b1by @localai-org-maint-bot in #11810 - chore: ⬆️ Update CrispStrobe/CrispASR to
78c545eb80409b91291642ddb23b3a6dc044fd34by @localai-org-maint-bot in #11811 - chore: ⬆️ Update 0xShug0/audio.cpp to
3497b7cc44753e2c141d8fe60ac42cec433e3281by @localai-org-maint-bot in #11818 - chore: ⬆️ Update mudler/vllm.cpp to
839ea1ceddb787778b6bd86a38a917a1aab74d8fby @localai-org-maint-bot in #11817 - chore: ⬆️ Update ggml-org/llama.cpp to
3466812d1f06728effe7c0f3c0671117f461672dby @localai-org-maint-bot in #11798 - chore: ⬆️ Update 0xShug0/audio.cpp to
f334cff70a68ea3d2e40d6638733e8c1ec434164by @localai-org-maint-bot in #11830 - chore: ⬆️ Update mudler/vllm.cpp to
6bf3abb580982f4fd2e4525ef37802ee0ce28981by @localai-org-maint-bot in #11828 - chore: ⬆️ Update ikawrakow/ik_llama.cpp to
caf7eae5282d840d77e9f91a56df7d2ef28fa612by @localai-org-maint-bot in #11842 - chore: ⬆️ Update NVIDIA/NeMo-Speech.cpp to
56b60d432f1731d6d5b28a4c5a31cbaf871daba1by @localai-org-maint-bot in #11846 - chore: ⬆️ Update CrispStrobe/CrispASR to
ff3945c94cab9191199a5d531a32c4e9535c094bby @localai-org-maint-bot in #11829 - chore: ⬆️ Update 0xShug0/audio.cpp to
c18b7f737aac0a2855e9f963a427498739ad40feby @localai-org-maint-bot in #11843 - chore: ⬆️ Update ggml-org/llama.cpp to
9cffdcc801582616250520966699cb5b25d28243by @localai-org-maint-bot in #11845 - chore(deps): bump the npm_and_yarn group across 1 directory with 2 updates by @dependabot[bot] in #11835
- chore: ⬆️ Update ikawrakow/ik_llama.cpp to
fe215a8ccdce6b844d2a3a3bbde08ae76a6284bfby @localai-org-maint-bot in #11864 - chore(deps): bump grpcio from 1.83.0 to 1.83.1 in /backend/python/coqui by @dependabot[bot] in #11856
- chore(deps): bump grpcio from 1.82.1 to 1.83.1 in /backend/python/rerankers by @dependabot[bot] in #11852
- chore(deps): bump grpcio from 1.83.0 to 1.83.1 in /backend/python/vllm by @dependabot[bot] in #11853
- chore: ⬆️ Update 0xShug0/audio.cpp to
8fe1ab805555c72aaf06d83828b523dfc6ed796dby @localai-org-maint-bot in #11861 - chore: ⬆️ Update ServeurpersoCom/omnivoice.cpp to
78ce63c90d36b94a765073e8cbe0d241fc114590by @localai-org-maint-bot in #11865 - chore: ⬆️ Update NVIDIA/NeMo-Speech.cpp to
69d7fd46b253043ed6649bf35d68407448912a8eby @localai-org-maint-bot in #11863 - chore: ⬆️ Update ggml-org/whisper.cpp to
52a939a2a762224e255d366c1182b2af4dd1a032by @localai-org-maint-bot in #11877 - chore(deps): bump grpcio from 1.82.1 to 1.83.1 in /backend/python/common/template by @dependabot[bot] in #11858
- chore: ⬆️ Update 0xShug0/audio.cpp to
cd98afb3264044aeff33618df486c81de1bee7e7by @localai-org-maint-bot in #11878 - chore(deps): bump the npm_and_yarn group across 1 directory with 3 updates by @dependabot[bot] in #11869
- chore: ⬆️ Update ggml-org/llama.cpp to
427291b5b34cd914a31b3fd3b61a68f6184f4b9fby @localai-org-maint-bot in #11866 - chore: ⬆️ Update CrispStrobe/CrispASR to
c6d4265ae2ee2b8931b09d7d25d5c65c75c36a41by @localai-org-maint-bot in #11862 - chore(deps): bump torch from 2.13.0+xpu to 2.14.0+xpu in /backend/python/common/template by @dependabot[bot] in #11859
- chore: ⬆️ Update NVIDIA/NeMo-Speech.cpp to
ffa38cb2408f1e832a36d46fef5e3e1e80d07e6cby @localai-org-maint-bot in #11887 - chore: ⬆️ Update ServeurpersoCom/omnivoice.cpp to
040c8b344d8c670ce1475194751d119b5ef82c78by @localai-org-maint-bot in #11890 - chore: ⬆️ Update 0xShug0/audio.cpp to
c41ffefb44c16345c0ac57a0030f4cd3b5fbc4b8by @localai-org-maint-bot in #11888 - chore: ⬆️ Update ggml-org/llama.cpp to
74a7c897f049c17e7080423aa2111776eff6ebbfby @localai-org-maint-bot in #11889 - chore: ⬆️ Update 0xShug0/audio.cpp to
f6277c1695a83cf388a8282c1c1a8757cf626f18by @localai-org-maint-bot in #11899 - chore: ⬆️ Update ggml-org/llama.cpp to
465e49b9cea78a68b9c244ffb48d0ee24a82873dby @localai-org-maint-bot in #11897 - chore: ⬆️ Update antirez/ds4 to
9ab705347c1775e7599ede7eb81a6255ec7dccb5by @localai-org-maint-bot in #11808 - chore: ⬆️ Update leejet/stable-diffusion.cpp to
d8fb10c02977c8ca999f3fb4e02df9ecf10f7ba6by @localai-org-maint-bot in #11898 - chore: ⬆️ Update ggml-org/llama.cpp to
67672dc5b76f8bc17785a19d3dc6d1463fc2902cby @localai-org-maint-bot in #11920 - chore: ⬆️ Update leejet/stable-diffusion.cpp to
d04e8950c1ec8d30248cbe996682b3182fb1adf6by @localai-org-maint-bot in #11919 - chore: ⬆️ Update mudler/depth-anything.cpp to
14f7461d1f704761a038ac9f50dbde8fdb7275e2by @localai-org-maint-bot in #11918 - chore: ⬆️ Update antirez/ds4 to
f62ca29a308724cde5bc99134ede19104b2a3260by @localai-org-maint-bot in #11917 - chore: ⬆️ Update CrispStrobe/CrispASR to
301acd87b036764973b8bfba71e0a21818036d33by @localai-org-maint-bot in #11916 - chore: ⬆️ Update 0xShug0/audio.cpp to
9c6a282337cc83f227cc10428867a478947706adby @localai-org-maint-bot in #11915 - chore: ⬆️ Update ggml-org/whisper.cpp to
c44b60b8053bbf2a5c1e014f11323fb3f2485177by @localai-org-maint-bot in #11937 - chore: ⬆️ Update antirez/ds4 to
6289c516273979173abbc062209a81dd3706b804by @localai-org-maint-bot in #11936 - chore: ⬆️ Update NVIDIA/NeMo-Speech.cpp to
a5b6953c4a579a2bbd1c0913ad8a85c2a4d99953by @localai-org-maint-bot in #11935 - chore: ⬆️ Update 0xShug0/audio.cpp to
05e508a70e3600b01454c647cdb122133ba8e64cby @localai-org-maint-bot in #11933 - chore: ⬆️ Update ikawrakow/ik_llama.cpp to
1a2a8604a6c6c6413c06bf9adfc2f64329af4366by @localai-org-maint-bot in #11932 - chore: ⬆️ Update ggml-org/llama.cpp to
f3f1a8f2760f28325a5ec20c05b171e5b7c83a29by @localai-org-maint-bot in #11934 - chore(deps): bump the npm_and_yarn group across 1 directory with 2 updates by @dependabot[bot] in #11957
- chore: ⬆️ Update 0xShug0/audio.cpp to
fa5aaac9266a98c68f8a5c9fcd1ba6ff65875416by @localai-org-maint-bot in #11955 - chore: ⬆️ Update ikawrakow/ik_llama.cpp to
3e416d7f5a9d4cc3195e8171dbf891541ca59c6aby @localai-org-maint-bot in #11952 - chore: ⬆️ Update ggml-org/llama.cpp to
434ddbbc0e30522e897670681e503b797c12b7c1by @localai-org-maint-bot in #11953 - chore: ⬆️ Update vllm-project/vllm cu130 wheel to
0.29.0by @localai-org-maint-bot in #11954 - chore: ⬆️ Update 0xShug0/audio.cpp to
efb04233dab73aeee4b2912042a90e7b36329061by @localai-org-maint-bot in #11964 - chore: ⬆️ Update ikawrakow/ik_llama.cpp to
3bb386eb68ffee0a5dc7db21da0735d594929eebby @localai-org-maint-bot in #11967 - chore: ⬆️ Update ggml-org/llama.cpp to
df03399b885831b2a1603b3abb0d8c156808e363by @localai-org-maint-bot in #11966 - chore: ⬆️ Update leejet/stable-diffusion.cpp to
b68d58624d227682eb4b95ef8bcf569cd1311eb5by @localai-org-maint-bot in #11965 - chore(deps): bump securego/gosec from 2.27.1 to 2.29.0 by @dependabot[bot] in #11854
- chore(deps): bump vllm from 0.28.0 to 0.29.0 in /backend/python/vllm by @dependabot[bot] in #11961
- chore(deps): bump LocalAGI and localrecall to v0.6.5 by @localai-org-maint-bot in #11985
- chore: ⬆️ Update leejet/stable-diffusion.cpp to
7f410a3793c5bba8eb198e962ce7a3d6095f9d89by @localai-org-maint-bot in #12012 - chore: ⬆️ Update CrispStrobe/CrispASR to
ab87fba65a4ade1ef30ccaa881458a0914aae557by @localai-org-maint-bot in #12011 - chore: ⬆️ Update 0xShug0/audio.cpp to
5ba81ac54fb071b835680973f8868546b4db372bby @localai-org-maint-bot in #12010 - chore: ⬆️ Update ggml-org/whisper.cpp to
1da4dc82fa7996d4edda05890dca65aeceaafd6dby @localai-org-maint-bot in #12009 - chore: ⬆️ Update OpenNMT/CTranslate2 ROCm wheel to
v4.8.2by @localai-org-maint-bot in #12004 - chore: ⬆️ Update ServeurpersoCom/omnivoice.cpp to
2ebbdcf281ce4b632e326011974da5a8b35c6b27by @localai-org-maint-bot in #12006 - chore: ⬆️ Update vllm-metal (darwin) to
v0.29.0by @localai-org-maint-bot in #12007 - chore: ⬆️ Update CrispStrobe/CrispASR to
a4db74cfd13c8674a7815608baaa505e673ce930by @localai-org-maint-bot in #12035 - chore: ⬆️ Update 0xShug0/audio.cpp to
ff1bcc4555ff99c4383329b0b21b52a18cc8b3cdby @localai-org-maint-bot in #12033 - chore: ⬆️ Update antirez/ds4 to
a04f46fa423e45712c8c7e430eff422479f314a3by @localai-org-maint-bot in #12013 - chore: ⬆️ Update mudler/vllm.cpp to
60990ee784101f74f6d1775575e9e89dfb26f73aby @localai-org-maint-bot in #12014 - chore: ⬆️ Update ggml-org/llama.cpp to
5f436dddb440a288ee5611d7d1eca564a6aca9f4by @localai-org-maint-bot in #12008 - chore: ⬆️ Update leejet/stable-diffusion.cpp to
42d6c0ab92fe6595776b28e3f7c8925db79b31f5by @localai-org-maint-bot in #12034 - chore: ⬆️ Update PrismML-Eng/llama.cpp to
d8f26eec76da6d09bb708bcba51ef64b8cd868a3by @localai-org-maint-bot in #12005 - chore: ⬆️ Update TheTom/llama-cpp-turboquant to
407f3237bfb3eeaff61546797de3d8c1a96be748by @localai-org-maint-bot in #12003 - chore: ⬆️ Update ikawrakow/ik_llama.cpp to
7434a014237e55b2e0d36edc2ace6acfb82fea1cby @localai-org-maint-bot in #12052 - chore: ⬆️ Update CrispStrobe/CrispASR to
97cea0919f1b5abfa2395f5791f88a05a0151223by @localai-org-maint-bot in #12054 - chore: ⬆️ Update 0xShug0/audio.cpp to
582a63ca8b6f76235f1b8636e734930218c78c05by @localai-org-maint-bot in #12056 - chore: ⬆️ Update mudler/vllm.cpp to
67c81102777296fdd506945860eb3030b9fad71aby @localai-org-maint-bot in #12053 - chore: ⬆️ Update antirez/ds4 to
9139e2ae58a41503968a500f36f75895c1ba63fcby @localai-org-maint-bot in #12062 - chore: ⬆️ Update PrismML-Eng/llama.cpp to
6ac5eb087009a762a3f61876f7dc152bc8600bf4by @localai-org-maint-bot in #12055 - chore: ⬆️ Update ggml-org/llama.cpp to
96ffdc41ceb055e1c2d3d96667ae6d9f0ccb710bby @localai-org-maint-bot in #12057 - chore: ⬆️ Update ServeurpersoCom/omnivoice.cpp to
3d341e55d365423476f8739d396b9a03d5277055by @localai-org-maint-bot in #12059 - chore: ⬆️ Update ggml-org/whisper.cpp to
1d549b3cecc2d98d76d4ddc2edca0d1512f5d7a0by @localai-org-maint-bot in #12060 - chore: ⬆️ Update leejet/stable-diffusion.cpp to
59c23bce0d82be3a922023ab811194f05b3e2faaby @localai-org-maint-bot in #12061 - chore: ⬆️ Update ServeurpersoCom/omnivoice.cpp to
3c30bdcfe6cc53121bd7a7b377bf8b55f526efa3by @localai-org-maint-bot in #12079 - chore: ⬆️ Update ikawrakow/ik_llama.cpp to
d85a1d7e91e5bc2f83a796542a8d5c436d32c99fby @localai-org-maint-bot in #12078 - chore: ⬆️ Update CrispStrobe/CrispASR to
89dfbc05d1466f6c7517986c6ff85179507431f8by @localai-org-maint-bot in #12077 - chore: ⬆️ Update 0xShug0/audio.cpp to
4af143229384fb6da3f373dc87de145ae954609bby @localai-org-maint-bot in #12073 - chore: ⬆️ Update ggml-org/whisper.cpp to
da54572229bcf64ba367d96c7ef15770376c4280by @localai-org-maint-bot in #12076 - chore: ⬆️ Update mudler/vllm.cpp to
b54db871e8478d0a3450132fa793d30cc336b321by @localai-org-maint-bot in #12074 - chore: ⬆️ Update ggml-org/llama.cpp to
38a5b42d9a3e82e0a586bcd1caed121f36c87a73by @localai-org-maint-bot in #12075 - chore: ⬆️ Update PrismML-Eng/llama.cpp to
7dffb158de30ebb8ef9d64f33c6b0b2d7c1e6313by @localai-org-maint-bot in #12072 - chore: ⬆️ Update antirez/ds4 to
8db1d1d155cb0400a86a86b9c62d0defb3a6148bby @localai-org-maint-bot in #12093 - chore: ⬆️ Update CrispStrobe/CrispASR to
ba3499e7c7f6013a73738cad530b252d59675f49by @localai-org-maint-bot in #12092
Other Changes
- docs(blog): add the 4.9 release post and its demo clips by @localai-bot in #11629
- docs: ⬆️ update docs version mudler/LocalAI by @localai-bot in #11643
- chore: ⬆️ Update ggml-org/llama.cpp to
d59d455fd8ea09e5a2e87ce2a9d668267ffb5ccdby @localai-bot in #11618 - chore: bump inference defaults from unsloth by @github-actions[bot] in #11654
- chore: ⬆️ Update 0xShug0/audio.cpp to
43001a7e0f452d80f4588e613f13332940dd4d3aby @localai-bot in #11649 - chore(model-gallery): ⬆️ update checksum by @localai-bot in #11645
- feat(xio): make copy buffer size configurable by @localai-bot in #11660
- fix(progress): stop status updates throttling downloads by @localai-bot in #11661
- fix(distributed): keep staging operations stable by @localai-bot in #11663
- feat(ui): edit scheduling rules in place by @localai-bot in #11667
- chore(model-gallery): ⬆️ update checksum by @localai-bot in #11665
- fix(distributed): keep model replicas config-consistent by @localai-bot in #11664
- chore: ⬆️ Update CrispStrobe/CrispASR to
74bb374a8cc74284348d76a0a6e944180fbe6b07by @localai-bot in #11650 - chore: ⬆️ Update 0xShug0/audio.cpp to
4d383be1bff107e823ffc19120dcb6c78d493c0fby @localai-bot in #11666 - chore: ⬆️ Update ggml-org/whisper.cpp to
233fe1fc9b48a09e361d3594520838ca266537feby @localai-bot in #11648 - chore(model-gallery): ⬆️ update checksum by @localai-bot in #11690
- chore: ⬆️ Update antirez/ds4 to
c1d4597a80e300b803dc642519718f2c999589daby @localai-bot in #11685 - chore(website): refresh the counters by @localai-bot in #11697
- fix(ci): bound Discord release summaries by @localai-org-maint-bot in #11695
- fix(oci): resume interrupted layer downloads by @localai-bot in #11688
- chore: ⬆️ Update CrispStrobe/CrispASR to
ae4474dd8306384a0e697183d863dfc52e69a2fbby @localai-bot in #11684 - chore: ⬆️ Update 0xShug0/audio.cpp to
288a2712316470847a730e55db9ac9e5062a2b03by @localai-bot in #11683 - feat(swagger): update swagger by @localai-bot in #11682
- chore(model gallery): 🤖 add 1 new models via gallery agent by @localai-bot in #11692
- chore(model-gallery): ⬆️ update checksum by @localai-bot in #11707
- chore: ⬆️ Update ikawrakow/ik_llama.cpp to
0ed847d3140baead542abe3e5e6fe841013e7340by @localai-bot in #11708 - chore: ⬆️ Update 0xShug0/audio.cpp to
d25ffac094a9d5a240940b4955ea79ad9b7b4c78by @localai-bot in #11710 - chore: ⬆️ Update ggml-org/llama.cpp to
f280b26983ad0fdb705a0d9ebf0503e76f2899b0by @localai-bot in #11646 - feat(gallery): add Ornith 1.5 397B variants by @localai-org-maint-bot in #11716
- chore: ⬆️ Update ikawrakow/ik_llama.cpp to
08b500b958a3f1102e6500e5c425e65517d6fb7eby @localai-bot in #11726 - chore: ⬆️ Update 0xShug0/audio.cpp to
c79e58899bf13db4d78fd06372da23cc13f55b28by @localai-bot in #11722 - chore: ⬆️ Update ggml-org/whisper.cpp to
978113305b2ead22249b881deafa131dc8884911by @localai-bot in #11711 - chore(model-gallery): ⬆️ update checksum by @localai-bot in #11730
- Fix flaky "tests-apple" CI job in modeladmin test suite by @mudler with @Copilot in #11717
- chore: ⬆️ Update ggml-org/llama.cpp to
eab8ee41f889ef7823af517e8098fb8a9b3cf601by @localai-bot in #11724 - chore: ⬆️ Update leejet/stable-diffusion.cpp to
50d640568388f876b0d63ee6ddb6bc86d997ec64by @localai-bot in #11725 - chore: ⬆️ Update mudler/vllm.cpp to
6738e0b4639199f3ff0998815e4d32bfa7fe5be2by @localai-bot in #11647 - chore(model-gallery): ⬆️ update checksum by @localai-bot in #11742
- chore: ⬆️ Update PrismML-Eng/llama.cpp to
312bb2a93ea2bf798333fa859614fbf913ecb9e2by @localai-bot in #11740 - chore: ⬆️ Update vllm-project/vllm cu130 wheel to
0.28.0by @localai-bot in #11741 - chore: ⬆️ Update ikawrakow/ik_llama.cpp to
ef40550042973817ac391ca95a2ff041f512257bby @localai-bot in #11743 - chore: ⬆️ Update ggml-org/llama.cpp to
925e1179947ea0c0ebfb0032df18af3a729822beby @localai-bot in #11744 - chore: ⬆️ Update 0xShug0/audio.cpp to
db21cbdd60f3d2ff62114bc863781ff8073ac39bby @localai-bot in #11746 - chore: ⬆️ Update mudler/depth-anything.cpp to
739992d10bf9472c46dcd4622b14d2b20766c58dby @localai-bot in #11758 - chore: ⬆️ Update ikawrakow/ik_llama.cpp to
7cff686d3732bfef5ce18bc4a6115fbceda29c14by @localai-bot in #11757 - chore: ⬆️ Update leejet/stable-diffusion.cpp to
be0e34480dada95f8ce9a021bbb95c5de85d67c7by @localai-bot in #11760 - chore: ⬆️ Update 0xShug0/audio.cpp to
17751c0e8c48a3d56dcf05eeb60464409ecc69ceby @localai-bot in #11759 - chore: ⬆️ Update ggml-org/llama.cpp to
e70802a01f03f0ed31a26338a5664796f3824371by @localai-bot in #11755 - fix(ui): size model fit against the cluster and move node labels into the selector by @localai-bot in #11765
- chore: ⬆️ Update ikawrakow/ik_llama.cpp to
15dddc60b3fc937a9e2a210359ecce392ccdf446by @localai-bot in #11767 - chore: ⬆️ Update antirez/ds4 to
8db89fe083ae4d17c9a2428ccd29803d3ae8f577by @localai-bot in #11768 - chore: ⬆️ Update ggml-org/llama.cpp to
d7bd3bfcad3e29c7e49fd26f38c79ee3e9a3fd6bby @localai-bot in #11770 - feat(distributed): key scheduling rules by a model alias by @localai-bot in #11771
- chore: bump inference defaults from unsloth by @github-actions[bot] in #11773
- chore: ⬆️ Update 0xShug0/audio.cpp to
89a0e9803380880305e9e1b83c93614f9df2c893by @localai-bot in #11769 - chore: ⬆️ Update mudler/vllm.cpp to
150b37852c123f7855fb219b37347572ca9427e7by @localai-bot in #11745 - feat(stablediffusion-ggml): build a ROCm variant by @localai-bot in #11774
- chore(model gallery): 🤖 add 1 new models via gallery agent by @localai-bot in #11777
- refactor(worker): iterate node label pairs with SplitSeq by @ginavalent in #11806
- fix(ci): stop the e2e teardown from failing a green suite by @localai-org-maint-bot in #11816
- chore(model-gallery): ⬆️ update checksum by @localai-org-maint-bot in #11831
- fix(ci): unbreak the e2e build and the darwin vllm-metal pin by @localai-org-maint-bot in #11849
- chore(model-gallery): ⬆️ update checksum by @localai-org-maint-bot in #11868
- chore(model gallery): 🤖 add 1 new models via gallery agent by @localai-org-maint-bot in #11880
- feat(swagger): update swagger by @localai-org-maint-bot in #11914
- chore(website): refresh the counters by @localai-org-maint-bot in #11902
- chore(model-gallery): ⬆️ update checksum by @localai-org-maint-bot in #11939
- chore(model-gallery): ⬆️ update checksum by @localai-org-maint-bot in #11968
New Contributors
- @localai-org-bot made their first contribution in #11631
- @DanielSwift1992 made their first contribution in #11686
- @leilei3167 made their first contribution in #11715
- @sz-po made their first contribution in #11736
- @blackd made their first contribution in #11750
- @ginavalent made their first contribution in #11806
- @FiloSpaTeam made their first contribution in #11802
- @aimansour made their first contribution in #11700
- @simpleqt made their first contribution in #11885
- @alex-mazzariol made their first contribution in #11749
- @seven7763 made their first contribution in #11310
- @QiuLsG made their first contribution in #11883
- @SomSamantray made their first contribution in #11652
- @devv-shayan made their first contribution in #11962
- @Nathan-Roll1 made their first contribution in #11998
- @quartzjer made their first contribution in #12045
Full Changelog: v4.9.0...v4.10.0
AI summary Authentication now deny-by-default, enhanced chat context compression
🎉 LocalAI 4.9.0 Release! 🚀
LocalAI 4.9.0 is out!
Thirteen days and 146 pull requests, spent on the parts of LocalAI you touch every day rather than on new engines. Authentication is now deny-by-default, chat gained end-to-end context compression, models and backends each have one canonical page instead of three, and vllm-cpp grew a video modality serving MiniMax-H3 with a real audio track.
Highlights:
- 🔐 Authentication is deny-by-default - every HTTP route requires credentials unless it appears in an explicit public registry. This closes a class of bypass in which unprefixed aliases such as
/moderations,/models,/backendsand/mcp/chat/completionsfell outside the old protected-prefix list. Reported by Naor Yaacov. - 🗜️ Chat context compression - opt-in per model, older complete turns are compressed through a LocalAI model before inference, preserving system prompts, the newest messages and whole tool-call units. Ratio and duration come back as response metadata and metrics.
- 🖥️ One page per resource -
/app/modelsnow owns Explore and Installed,/app/backendsowns Catalog and Installed, and the nested Host view is gone. Old/app/managebookmarks still work. - 🎬 MiniMax-H3 video generation -
vllm-cppopens a second engine handle for the H3 checkpoint set and renders video and audio jointly, so the MP4 arrives with a real AAC track. Ask for speech in the prompt and the model lip-syncs it. - 🗣️ Qwen3-TTS on llama.cpp - text-to-speech on the full accelerator matrix already shipped for text generation (CUDA, ROCm, SYCL, Vulkan, Metal, L4T), using upstream's own GGUF conversion.
- 🧭 KNN as a first-class router - similarity-weighted voting over a curated, persisted corpus of labelled prompts. No classifier model, and a prompt unlike anything labelled is treated as undecidable rather than guessed.
- 📊 Global admission control and live backend traces - process-wide HTTP admission bounds, in-flight backend operations are represented while they run, and the UI links straight to their logs.
- 🕵️ Reversible PII pseudonyms - masked values become request-scoped deterministic pseudonyms (
EMAIL_001) and are restored if the backend echoes them, across JSON and SSE tokens split over writes. - 📦 Parallel Hugging Face downloads - snapshot materialization runs up to N whole-file transfers at once, so a repository split into many shards stops spending its wall clock in per-file latency.
- 🖧 Cold model loads are durable jobs - the per-model advisory lock no longer spans a multi-GB transfer, which had made a 35.7 GB load look permanently broken from the operator's seat while staging progressed normally underneath.
- 🎮 vllm-cpp covers the cards you own - CUDA builds went from one or two architectures to eight on amd64 and five on arm64, picking up A100, L4, 4090, H100/H200, B200, Jetson Orin and Jetson Thor.
Plus a single shared WebRTC UDP port for Realtime, Metal actually enabled in the macOS Stable Diffusion and Parakeet builds, backend crash diagnostics at the default log level, and Portuguese (Brazil) and Indonesian UI translations.
Models and backends each get one canonical page, with Explore and Installed as views rather than separate destinations.
📊 This release in numbers
| Pull requests merged | 146 |
| Commits | 148 |
| Files changed | 420 (+27,788 / -4,670) |
| Development window | 13 days (2026-08-07 to 2026-08-20) |
| Human contributors | 10, of whom 3 first-time |
| Gallery entries | 1,622 to 1,707 (+85) |
Where the work landed:
| Area | Change |
|---|---|
core/ |
+17,193 / -4,187 across 279 files |
gallery/ |
+3,808 / -62 |
backend/ |
+3,298 / -177 across 60 files |
pkg/ |
+1,180 / -113 across 31 files |
swagger/ |
+1,095 / -5 |
docs/ |
+952 / -98 across 23 files |
📌 TL;DR
| Area | Summary |
|---|---|
| 🔐 Auth by default | Every method and path requires credentials unless listed in an explicit public registry. API instructions, Swagger GETs, the LocalAI well-known document, and the health, login, OAuth, SPA, asset, branding and node registration-token bootstrap flows stay public. Migration: with database auth or legacy API keys configured, /version and generated audio, image, video and 3D URLs now require credentials. Embedded deployments can add narrow prefixes through ApplicationConfig.PathWithoutAuth, and the legacy GET exemption flags remain as explicit compatibility overrides. |
| 🗜️ Context compression | Opt-in per-model compression config. Older complete turns are compressed through a configured LocalAI model before inference, after PII filtering and Assistant/MCP prompt injection (including later MCP iterations). Leading system/developer prompts, the newest messages and complete tool-call/result units are preserved; tool schemas and completion headroom are accounted for with a conservative offline token bound. Metadata rides non-streaming responses and streaming usage trailers, with event, ratio and duration metrics exported. Disabled by default; cloud-proxy passthrough is rejected (translate mode works). |
| 🖥️ Unified lifecycle UI | /app/models owns Explore and Installed with URL-backed search, state and selection; /app/backends owns Catalog and Installed while keeping variants, development builds and target-node scope. Explore offers capability-aware Open and Manage installation; destructive model controls stay in Installed. Operate Overview shows host capacity from its shared summary poller, and the nested Host destination is removed. /app/manage redirects while preserving legacy query state. No API change. |
| 📥 Import form rebuild | The import page moves to page--medium with a work column and the format reference beside it rather than behind a closed chevron. The source field is the hero and carries its own Import button, which removes the aria-hidden submit that existed only because the real action sat outside the <form>. Simple and Advanced modes are gone (about 80% the same surface); the real distinction, a source or YAML, is now two tabs. Also fixes two class bugs: a primary button with no className at all falling through to browser chrome, and class="btn btn-primary fas fa-save fa-upload" setting Font Awesome as the button's own font while two icons fought over one ::before. |
| 🎬 MiniMax-H3 video | vllm-cpp over vllm.cpp ABI v12. A second engine handle loads the H3 checkpoint set (the DiT is parameters.model, the text encoder and two VAEs are named in options:), and GenerateVideo renders video and audio jointly into an MP4 with a real AAC track. The DiT partition is declared, not detected: community quantizations strip the release metadata and the FL2VA and Ref2VA DiTs are byte-structurally identical, so checkPartitionConditioning refuses a reference-conditioned FL2VA request before the engine runs (it would otherwise render for hours and return a coloured lattice). ffmpeg comes from the host: libvllm composes the mux argv and spawns nothing. New gallery entry minimax-h3-fl2va-q4. |
| 🗣️ Qwen3-TTS on llama-cpp | TTS through the llama-cpp backend on CUDA, ROCm, SYCL, Vulkan, Metal and L4T, using upstream's GGUF conversion. Implemented as a slot-based SERVER_TASK_TYPE_TTS task, which is the concurrency-safe integration given that server_context owns the llama_context and runs the slot scheduler on its own thread. Carries the still-draft upstream server hunks as patches/0002-add-server-task-type-tts.patch (delete on merge of ggml-org/llama.cpp#26603). Gallery: qwen3-tts-llamacpp and qwen3-tts-llamacpp-q4. The existing qwen3-tts-cpp backend is untouched and remains a separate path. |
| 🧭 KNN routing | classifier: knn routes by similarity-weighted voting over labelled example prompts, so no classifier model is needed and label knowledge lives in a corpus you seed and curate. Entries below knn.similarity_threshold cannot vote; when none clears it the router takes the fallback, and nearest_similarity is recorded on decisions and fallbacks alike. One JSONL file per router under <data path>/router-corpus is the source of truth, with the in-memory index rebuilt at classifier build time and entries re-embedded when the embedding model changed. Corpus input is API-only by design: POST /api/router/{name}/corpus, GET .../corpus/stats (label counts only, texts are never returned), DELETE .../corpus, admin-gated and exposed as MCP tools. |
| 📊 Admission and traces | Process-wide HTTP admission control, bounding what was previously only per backend. Backend operations are represented while in flight, and running backend traces surface in the UI with immediate log links. |
| 🕵️ PII pseudonyms | Opt-in pii.reverse_in_response. Masked request values become unique deterministic pseudonyms within the request (EMAIL_001, EMAIL_002) and are restored if the backend returns them, including SSE tokens split across response writes. Substitution maps are request-local and never persisted. Irreversible [REDACTED:...] remains the default. |
| 📦 Parallel HF downloads | DownloadFilesWithConcurrency runs up to N whole-file transfers through an errgroup with SetLimit. Single files are never split, so .partial resume and per-file SHA verification are untouched, and the two non-artifact callers keep sequential ordering and fail-fast behaviour through a limit-of-1 wrapper. completedBytes became an atomic.Int64 (the race detector reported three races otherwise) and the caller's status callback stays serialized. |
| 📞 Realtime WebRTC port | --web-rtc-udp-port / LOCALAI_WEBRTC_UDP_PORT reuses one Pion ICE UDP mux across Realtime calls, with bind failures surfaced through signaling and container/firewall setup documented. The follow-up fix keeps LOCALAI_WEBRTC_ICE_INTERFACES effective when a fixed port is set, which had been silently ignored: a wildcard mux made pion enumerate every interface itself, handing browsers unroutable 172.x candidates that dropped once ICE consent checks failed. |
| 🖧 Durable cold loads | The per-model advisory lock is a dedup decision measured in milliseconds, not a transfer's lifetime. Cold loads now run as durable jobs instead of holding it across backend install, multi-GB staging and checkpoint load, and WithLockCtx now defends against statement_timeout as well as lock_timeout (both abort the same blocking pg_advisory_lock, only the latter was overridden). |
| 🎮 vllm-cpp CUDA coverage | amd64 goes from 120a;121a to 80;86;89;90a;100a;103a;120a;121a, arm64 from 121a to 87;90a;100a;110;121a, split by where the silicon exists. An unlisted card did not run slower, it died at the first request with no kernel image is available for execution on the device, long after install reported success. The CUDA 13 guard now covers both branches, and Triton-AOT stays on. |
| 🌍 Two new languages | Portuguese (Brazil), a complete 14-namespace translation at full key parity with en/, and Indonesian for the admin, media and navigation surfaces. |
| 🧠 Models | 85 new gallery entries: Qwen3.8 (9B, 27B, Ridge and small variants), Gemma 4 agentic and Scotoma 2, DeepSeek V4 Pro 0813, Ling 3.0 Flash, Nemotron 3.5 Lightning 30B, Tess 4 27B, Ornith 1.0 and 1.5 9B, LFM2.5 230M and VL 1.6B, HunyuanOCR and OvisOCR2, Higgs Audio v3 TTS, MiniMax-H3 Ref2VA, plus vllm.cpp text-generation entries and a first Carbon genomics family. |
🚀 New Features & Major Enhancements
🔐 Authentication now denies by default
The previous classifier gated selected API-style paths by prefix. Anything whose path was not on that list was public, which meant unprefixed aliases (/mcp/chat/completions, /moderations, /models, /backends, /import-model) could bypass global authentication, and any newly registered route inherited the same weakness by default.
The middleware is now method-aware and denies by default: a route is public only if its method and path appear in an explicit public registry. What stays public is the set required to bootstrap and to be discoverable: API instructions, Swagger GET routes, the LocalAI well-known document, and the health, login, OAuth, SPA, asset, branding and node registration-token flows. Whole-router coverage is asserted in tests, so a new route cannot become public by omission.
Migration impact. When database authentication or legacy API keys are configured, /version and generated audio, image, video and 3D URLs now require credentials. Embedded deployments can still add narrow prefixes via ApplicationConfig.PathWithoutAuth, and the legacy GET exemption flags remain available as explicit compatibility overrides.
Thanks to Naor Yaacov for reporting this class of authentication bypass.
🔗 PRs: #11602
🗜️ End-to-end context compression
A long conversation eventually stops fitting. Compression is opt-in per model, and when enabled it compresses older complete turns through a configured LocalAI model before inference rather than truncating them away.
What it will not touch: leading system and developer safety prompts, the newest messages, and complete tool-call/result units, which are kept whole so a compressed history never leaves a call without its result. It runs after PII filtering and after Assistant/MCP prompt injection, including on later MCP iterations, so what gets compressed is the prompt that would actually have been sent. Tool schemas and completion headroom are accounted for with a conservative offline token bound.
Compression metadata is exposed in non-streaming responses and in streaming usage trailers, and compression events, ratios and durations are exported as metrics. Cloud-proxy passthrough configurations reject compression because LocalAI cannot safely rewrite an opaque provider payload; translate mode is supported. A late failure in an already-started stream is returned as an in-band SSE error followed by [DONE].
🔗 PRs: #11556
🖥️ One canonical page per resource
Models had a gallery and a separate Host management surface. Backends had a nested Host view for installed binaries. Between them it was not obvious where a resource lived, and the common lifecycle actions sat one level deeper than they needed to.
Each resource now has one page. /app/models owns Explore and Installed, /app/backends owns Catalog and Installed, both with URL-backed search, state and selection, and backends keep their variants, development builds and target-node scope. Explore presents capability-aware Open and Manage installation actions while destructive model controls stay in Installed. Operate Overview reads host capacity from its shared summary poller, and the nested Host destination is removed. Narrow list/detail views restore focus to the originating row when you come back from a detail view.
Nothing in the API changed, and existing /app/manage bookmarks keep working through a replace redirect that preserves legacy model and backend query state.
Catalog and Installed as views of one page, with target-node scope intact.
🔗 PRs: #11548
📥 The import form, rebuilt
The import page had taken the new palette but kept its old layout: a 760px column with the primary action detached from the form it submits. Two of its problems were outright bugs.
ImportModel.jsx:808 carried no className at all, so the page's single most important control fell through to the user-agent button, with system chrome, system font, the wrong radius and no design-system focus ring. Next to it, class="btn btn-primary fas fa-save fa-upload" set Font Awesome as the button's own font family, which its label text inherited, while fa-save and fa-upload fought over one ::before.
The layout moves to page--medium with a work column and the format reference beside it, since that reference answers the only question a first-time admin has and used to sit behind a chevron that was closed by default. Below 1024px it becomes a disclosure instead of disappearing. The source field is the hero, monospace because it holds something you paste, and it carries its own Import button, which removes the aria-hidden submit that existed only to compensate for the real action sitting outside the <form>. Simple and Advanced modes are gone: they were about 80% the same surface, and the overlap cost a mode switch, a localStorage key and a three-button Keep/Discard/Cancel dialog whose only job was protecting state the switch would have hidden. What genuinely differs is the kind of input, which is now two tabs: a source, or YAML. The size and VRAM estimate reports under the field that produced it instead of as a banner above the page header.
A follow-up swept the same class of bug across the rest of the UI: eight header controls on seven pages had two or three elements' classes collapsed into one string.
One form, a collapsible options panel, and the format reference where you can read it.
🎬 MiniMax-H3: video and audio, jointly
vllm.cpp's stable C ABI grew a video slice at v12, and vllm-cpp now serves two things. Text generation is unchanged. When a model config declares the H3 checkpoint set, Load opens a video engine instead and GenerateVideo renders a clip through LocalAI's existing /video endpoint, with video and audio generated jointly, so the MP4 comes back with a real AAC track rather than silent. Ask for speech in the prompt and the model lip-syncs it.
Three things shape the integration:
The video engine is a second handle, not a mode of the first. H3 is not a model directory. The DiT, the text encoder and two VAEs are separate artifacts, and vllm.cpp's two loaders refuse each other's checkpoints. parameters.model is the DiT; the rest of the set is named in options:.
The DiT partition is declared, not detected. The FL2VA DiT serves t2va and fl2va; ref2va is a different checkpoint. Community GGUF and NVFP4 quantizations strip the release metadata and the two DiTs are byte-structurally identical, so the engine refuses to generate until it is told which it has. Handing reference conditioning to an FL2VA DiT renders for hours and returns a coloured lattice over the frame, so checkPartitionConditioning rejects that combination before the engine is ever called.
ffmpeg comes from the host. libvllm writes the frames and the WAV and composes the mux argv, then spawns nothing, which is a deliberate upstream process boundary. The backend substitutes argv[0] and execs it.
Gallery entries minimax-h3-fl2va-q4 (Q4_K_M FL2VA set) and minimax-h3-ref2va-q4 ship with it.
🗣️ Qwen3-TTS through the llama.cpp backend
Qwen3-TTS now runs on the llama-cpp backend, which means text-to-speech on the same accelerator matrix already shipped for text generation (CUDA, ROCm, SYCL, Vulkan, Metal, L4T) using upstream's own GGUF conversion.
grpc-server.cpp is an adapter over llama.cpp's shared server_context, which owns the llama_context and runs the slot scheduler on its own thread, so a gRPC handler driving the gen-audio loop itself would race that scheduler. Making TTS a slot-based SERVER_TASK_TYPE_TTS task is the concurrency-safe integration, following the 0001-add-server-task-type-score.patch precedent already in the tree. llama.cpp merged Qwen3-TTS in ggml-org/llama.cpp#26254; the server plumbing in #26603 is still a draft, so it is carried as patches/0002-add-server-task-type-tts.patch and should be deleted once that merges. disable-tts-task.sh keeps turboquant and bonsai compiling, since they copy grpc-server.cpp into forks without our patches.
Both paths were verified end to end on CPU returning valid 24 kHz mono 16-bit WAV containing real speech, measured rather than eyeballed. Gallery entries: qwen3-tts-llamacpp and qwen3-tts-llamacpp-q4.
The existing qwen3-tts-cpp backend over qwentts.cpp is untouched. This is a second, independent path, not a replacement.
🔗 PRs: #11392
🧭 KNN as a first-class router
classifier: knn promotes KNN search from a cache for the classifier to a primary request router. Unlike score or colbert it needs no classifier model: label knowledge lives in a corpus of labelled example prompts that you seed and curate through the admin API, so routing decisions are deterministic, auditable, and grounded in graded experience rather than a model's opinion.
There is an explicit epistemic gate. Corpus entries below knn.similarity_threshold cannot vote, and when none clears it the classifier activates no labels and the router takes the fallback: a prompt unlike all labelled experience is treated as undecidable, not guessed. Decisions record nearest_similarity, on fallback rows too, so you can see how far the nearest labelled experience actually was, and the Routing tab explains out-of-corpus fallbacks and shows per-label corpus counts.
Persistence is one JSONL file per router under <data path>/router-corpus holding text, labels, vector and embedder fingerprint. That file is the source of truth; the local-store index is rebuilt from it at classifier build time and stays a pure in-memory index, and entries recorded under a different embedding model re-embed on load. This also corrects the docs' claim that local-store collections persist: the embedding cache never survived restarts and still does not, while the corpus does.
Corpus input is API-only by design, since entries may contain example user content: POST /api/router/{name}/corpus seeds (labels validated against declared policies, embedded server-side, indexed immediately), GET .../corpus/stats inspects and returns label counts only (entry texts are never returned by any surface), and DELETE .../corpus wipes. All admin-gated like the sibling router endpoints and exposed as MCP tools.
🔗 PRs: #10652
📊 Global admission control and running backend traces
Admission control existed per backend, which left nothing bounding the process as a whole, and the traces list could grow without limit. HTTP admission is now bounded process-wide.
Alongside it, backend operations are represented while they are still in flight rather than only once they finish, and running backend traces surface in the UI with immediate links to their logs, so an operation that is taking too long is something you can look at instead of something you wait out.
Operations are represented while they are still running, not only once they finish.
🔗 PRs: #11560
🕵️ PII pseudonyms that survive the round trip
The PII middleware masked values irreversibly, which is right for logs and wrong for a conversation: a model that is handed [REDACTED:EMAIL] twice cannot tell whether it saw one address or two, and anything it says about them comes back unusable.
Opt-in pii.reverse_in_response turns masked request values into unique deterministic pseudonyms within the request (EMAIL_001, EMAIL_002) and restores them if the backend returns them. Restoration handles normal JSON and SSE tokens split across response writes. Substitution maps stay request-local and are never persisted. Irreversible [REDACTED:...] remains the default.
🔗 PRs: #11272
📦 Bounded parallel Hugging Face downloads
Snapshot materialization fetched every file through the sequential executor, so a repository split into many shards spent most of its wall clock in per-file request latency rather than moving bytes.
DownloadFilesWithConcurrency now runs up to N whole-file transfers at once through an errgroup with SetLimit. Only whole files run in parallel: a single file is never split, so the .partial resume machinery and the per-file SHA check are untouched. The two non-artifact callers (core/gallery/models.go and core/config/model_config_loader.go) keep exactly their previous behaviour through a wrapper passing a limit of 1, so tasks still run in slice order and the first failure still returns before any later task starts.
Two consequences of the parallel path are worth knowing: completedBytes is now an atomic.Int64, which is not a precaution (with a plain int64 the race detector reports three races), and the caller's status callback is serialized to preserve the guarantee the sequential path gave it implicitly. AfterDownload is deliberately not serialized, because it does the verify-and-promote work that the parallelism exists to overlap.
🔗 PRs: #11162
📞 Realtime WebRTC on one UDP port
--web-rtc-udp-port / LOCALAI_WEBRTC_UDP_PORT reuses a single Pion ICE UDP mux across Realtime WebRTC calls, so a container or firewall needs one rule rather than a range. UDP bind failures are surfaced through signaling instead of failing opaquely, and the container and firewall setup is documented.
A follow-up closed the gap it opened. LOCALAI_WEBRTC_ICE_INTERFACES was silently ignored whenever a fixed UDP port was set, which is exactly the combination an operator reaches for: pinning a port to write a firewall rule and restricting interfaces to keep unreachable docker0 and veth addresses out of the candidate list usually go together. A wildcard mux made pion derive candidate addresses by enumerating interfaces itself with a nil filter, so on a host with docker bridges the browser received 172.18.0.1, 172.17.0.1, 10.10.10.1 and friends, connected on a good pair, then dropped when ICE consent checks failed on the others.
🖧 Cold model loads run as durable jobs
On a two-replica frontend, loading a 35.7 GB GGUF onto a newly added Jetson Thor worker made the model permanently unloadable from the operator's seat, while staging was in fact progressing normally underneath. Replica A held the per-model advisory lock through roughly twenty minutes of transfer; replica B blocked on pg_advisory_lock for the same model and was killed at 60s by the role's statement_timeout, and every UI retry reproduced it.
Two defects sat behind that one symptom. The lock's lifetime was the transfer's lifetime, with Route wrapping backend install, multi-GB staging and checkpoint load in advisorylock.WithLockCtx, which turns a millisecond dedup decision into a cluster-wide outage for that model. And WithLockCtx overrode lock_timeout but not statement_timeout, though both abort the same blocking call.
Cold loads now run as durable jobs, so the lock is held only for the decision it exists to make.
🔗 PRs: #11514
🎮 vllm-cpp builds for the cards people own
The vllm-cpp CUDA images were built for Blackwell only: 120a;121a on amd64 and 121a alone on arm64, out of the ten architectures vllm.cpp's own release archive builds.
What makes it worth calling out is the failure mode. An unlisted card is not slower, it dies at the first request with no kernel image is available for execution on the device, long after local-ai backends install vllm-cpp reported success. This was found on a Jetson Thor node that had the backend installed and could serve nothing.
| before | after | |
|---|---|---|
| amd64 | 120a;121a |
80;86;89;90a;100a;103a;120a;121a |
| arm64 | 121a |
87;90a;100a;110;121a |
The split follows where the silicon exists: Jetson (87 Orin, 110 Thor) is arm64-only, desktop 120a is amd64-only, and 90a/100a are on both because of the SBSA parts. That adds A100, A10/3090, L4/4090/RTX 6000 Ada, H100/H200, B200, B300, Jetson Orin and Jetson Thor. The CUDA 13 guard now covers both branches rather than amd64 alone, and Triton-AOT stays on.
🔗 PRs: #11512
🌍 Portuguese (Brazil) and Indonesian
A complete pt-BR translation of the WebUI: 14 namespaces at full key parity with en/ including modelEditor.json, with every i18next interpolation variable and _one/_other plural key preserved. Indonesian covers the admin, media and navigation strings. Both keep brand, model and technical identifiers untranslated, matching existing locale conventions.
🧰 Smaller features worth knowing about
- Metal is actually on in two macOS backends.
stablediffusion-ggmlgated its Metal flags on anOS=Darwinvariable the runner never defines, so the Darwin workflow'sBUILD_TYPE=metalproduced a build withoutGGML_METAL_EMBED_LIBRARY=ON, shipping a runtime source path that failed to exposekernel_mul_mv_ext_bf16_f32_r1_5.parakeet-cppnever forwardedBUILD_TYPE=metaltoPARAKEET_GGML_METALat all: on an M1 Air the same five-minute sample went from 82.57s to 50.18s with byte-identical output. A dry-run build-contract test now guards the Stable Diffusion flags. - Backend crashes say why. An unexpected runtime exit logged its stderr only at debug level, so at default log level operators saw an exit code and nothing else. The final non-empty stderr line now rides the unexpected-exit warning, and a failed gRPC readiness preserves the process exit code plus the last stderr diagnostic bounded to 4 KiB.
- MCP servers stay visible when they fail. Model-level MCP servers disappeared from Chat whenever connection setup or tool discovery failed, hiding container DNS, routing and VPN reachability problems. They now stay listed as disabled error rows with a per-server error, discovery retries while safely closing partially created sessions, and the model editor documents the expected
mcp.remoteandmcp.stdioformats. - Checksum mismatches retry. A post-download SHA mismatch is now treated as a transient transfer failure: LocalAI removes the mismatched partial and lets the bounded planner retry a stale or corrupted CDN response, while still refusing unverified bytes. Direct
URI.DownloadFilecallers still see the mismatch immediately. - Audio transform rejects the wrong contract. The audio-transform WebSocket accepted
realtime_audiomodels and opened the frame-basedAudioTransformStreamRPC, which failed after the handshake withNotImplementedErrorfor any-to-any models likeliquid-audio. The use case is now validated before the backend loads, and any-to-any callers are pointed at the OpenAI Realtime API. - Invalid preload JSON names itself.
PRELOAD_MODELS/--preload-modelsnow identifies which input was invalid and rejects non-array top-level values (including booleans andnull) with the expected shape in the error.
🔗 PRs: #11531, #11492, #11532, #11447, #11495, #11536, #11565, #11434
🐛 Bug Fixes (recap)
fix(auth): protect HTTP routes by default - #11602fix(distributed): run cold model loads as durable jobs instead of holding the advisory lock - #11514fix(vllm-cpp): build every CUDA architecture the platform can host - #11512fix(realtime): keep the ICE interface allow-list working with a fixed UDP port - #11466fix(stablediffusion): embed Metal library - #11531fix(parakeet-cpp): enable Metal in macOS builds - #11492fix(model): report backend crash diagnostics - #11532fix(model): surface backend startup exits - #11447fix(downloader): retry checksum mismatches - #11536fix(audio): reject incompatible transform streams - #11565fix(gallery): parse harmony output of gpt-oss-* models correctly - #11518fix(gallery): identify invalid preload JSON - #11434fix(gallery): repair DeepSeek V4 fallback - #11480fix(gallery): correct Higgs Audio v3 checksum - #11459fix(fish-speech): preserve ROCm PyTorch - #11568fix(vllm): align Intel basekit runtime to oneAPI 2025.3.2 - #11437fix(kokoros): add the missingupscale_imagestub to the Backend trait impl - #11414fix: show MCP connection errors in the UI - #11495fix(ui): unmerge the class strings that left buttons in browser chrome - #11462fix(ui): keep agent import action visible - #11488fix: wrap long TTS request text instead of widening the page - #11576
🧠 Models
85 new gallery entries this cycle, taking the index from 1,622 to 1,707.
Text generation: Qwen3.8 in 9B, 27B, Ridge and small variants, Gemma 4 agentic and Gemma 4 Scotoma 2, DeepSeek V4 Pro 0813, Ling 3.0 Flash, Nemotron 3.5 Lightning 30B, Tess 4 27B, Ornith 1.0 and 1.5 9B, Muse Glimmer 30B, Grug 12B, BigBang v1, Genesis Hermes V7, TwIL-LM3, BTL-4 Compact, XYZ Aquila mini, North Mini Code, AREX Turbo, Fara1.5 4B, MiniCPM5 1B Q8, Shieldstral 1.0 3B and UI-Mate 9B.
Vision and OCR: HunyuanOCR, OvisOCR2, LFM2.5 VL 1.6B, and LFM2.5 230M alongside it.
Audio and video: Higgs Audio v3 TTS, the Qwen3-TTS llama.cpp entries, and the MiniMax-H3 FL2VA and Ref2VA video sets.
Also a first Carbon genomics family, and text-generation entries for the vllm-cpp backend.
🔗 PRs: #11622, #11603, #11599, #11598, #11594, #11584, #11573, #11571, #11561, #11559, #11557, #11552, #11551, #11549, #11547, #11540, #11533, #11526, #11519, #11511, #11490, #11479, #11478, #11477, #11458, #11456, #11455, #11449, #11446, #11443, #11441, #11439, #11438, #11435
👒 Dependencies
Submodule and pin bumps this cycle:
| Project | Bumps |
|---|---|
| CrispStrobe/CrispASR | 10 |
| ikawrakow/ik_llama.cpp | 9 |
| vllm-metal (darwin) | 7 |
| mudler/vllm.cpp | 6 |
| 0xShug0/audio.cpp | 5 |
| ggml-org/llama.cpp | 4 |
| ggml-org/whisper.cpp | 3 |
| leejet/stable-diffusion.cpp | 2 |
| antirez/ds4, mudler/parakeet.cpp, mudler/depth-anything.cpp, NVIDIA/NeMo-Speech.cpp, vllm-project/vllm cu130 wheel | 1 each |
Plus golang.org/x/net to v0.55.0, vllm 0.26.0 and transformers >=5.15.0 in the Python backends, sentence-transformers 5.7.0, packaging 26.3, dompurify 3.4.13, the Kokoros source pin, inference defaults refreshed from unsloth, and seven gallery checksum refreshes.
📖 Documentation
New pages for context compression and vllm-cpp, a substantially expanded middleware page covering the PII pseudonym and compression surfaces, and a rewritten authentication page documenting the public and protected route surfaces after the deny-by-default change.
Video generation gained the MiniMax-H3 setup, text-to-audio the Qwen3-TTS llama.cpp path, distributed mode the durable cold-load behaviour, MCP the configuration formats and container networking implications, and Realtime the shared UDP port and firewall guidance. Embeddings, model gallery, backends and API discovery all picked up corrections, and the broken stars counter came out of the site.
🙌 New Contributors
- @tom-mi made their first contribution in #11518
- @kassane made their first contribution in #11427
- @fieryWaters made their first contribution in #11492
Thanks also to @richiejp, @jimmykarily, @Dennisadira, @dedyf5, @ALameLlama and @walcz-de, and to Naor Yaacov for the authentication bypass report.
What's Changed
Breaking Changes 🛠
- fix(auth): protect HTTP routes by default by @localai-bot in #11602
Bug fixes 🐛
- fix(kokoros): add missing
upscale_imagestub to Backend trait impl by @mudler with @Copilot in #11414 - fix(gallery): identify invalid preload JSON by @localai-org-maint-bot in #11434
- fix(vllm): align Intel basekit runtime by @localai-org-maint-bot in #11437
- fix(model): surface backend startup exits by @localai-org-maint-bot in #11447
- fix(ui): unmerge the class strings that left buttons in browser chrome by @localai-bot in #11462
- fix(realtime): keep the ICE interface allow-list working with a fixed UDP port by @jimmykarily in #11466
- fix(parakeet-cpp): enable Metal in macOS builds by @fieryWaters in #11492
- fix: Show MCP connection errors in the UI by @richiejp in #11495
- fix(vllm-cpp): build every CUDA architecture the platform can host by @localai-bot in #11512
- fix (gallery): Parse harmony output of gpt-oss-* models correctly (#8037) by @tom-mi in #11518
- fix(ui): keep agent import action visible by @localai-org-maint-bot in #11488
- fix(stablediffusion): embed Metal library by @localai-org-maint-bot in #11531
- fix(model): report backend crash diagnostics by @localai-org-maint-bot in #11532
- fix(distributed): run cold model loads as durable jobs instead of holding the advisory lock by @localai-bot in #11514
- fix(downloader): retry checksum mismatches by @localai-org-maint-bot in #11536
- fix(fish-speech): preserve ROCm PyTorch by @localai-org-maint-bot in #11568
- fix(audio): reject incompatible transform streams by @localai-org-maint-bot in #11565
- fix: tts text wrap by @ALameLlama in #11576
Exciting New Features 🎉
- feat(modelartifacts): support bounded parallel Hugging Face file downloads by @Dennisadira in #11162
- feat(i18n): add pt-BR translation by @kassane in #11427
- feat(vllm-cpp): serve MiniMax-H3 video+audio generation by @localai-bot in #11424
- feat(pii): restore request-scoped pseudonyms by @localai-org-maint-bot in #11272
- feat(llama-cpp): serve Qwen3-TTS through the llama.cpp backend by @localai-bot in #11392
- feat(realtime): add shared WebRTC UDP port by @localai-org-maint-bot in #11436
- feat(ui): rebuild the import form on the restyled design language by @localai-bot in #11461
- i18n(id): translate admin, media, and nav UI strings to Indonesian by @dedyf5 in #11493
- feat(ui): unify model and backend lifecycle by @localai-bot in #11548
- feat: bound global admission and expose running backend traces by @richiejp in #11560
- feat(router): make KNN a first-class classifier with a persisted, curated corpus by @richiejp in #10652
- feat(chat): add end-to-end context compression by @localai-org-maint-bot in #11556
🧠 Models
- feat(gallery): add Grug 12B variants by @localai-org-maint-bot in #11438
- feat(gallery): add Shieldstral 1.0 3B variants by @localai-org-maint-bot in #11441
- feat(gallery): add Genesis Hermes V7 variants by @localai-org-maint-bot in #11435
- feat(gallery): add minimax-h3-ref2va-q4 - the Ref2VA partition, verified coherent by @localai-bot in #11439
- feat(gallery): add BigBang v1 variants by @localai-org-maint-bot in #11443
- feat(gallery): add Gemma 4 Scotoma 2 variants by @localai-org-maint-bot in #11458
- feat(gallery): add Muse Glimmer 30B variants by @localai-org-maint-bot in #11446
- feat(gallery): add Higgs Audio v3 TTS by @localai-org-maint-bot in #11456
- feat(gallery): add BTL-4 Compact by @localai-org-maint-bot in #11449
- feat(gallery): add XYZ Aquila mini variants by @localai-org-maint-bot in #11455
- fix(gallery): correct Higgs Audio v3 checksum by @localai-org-maint-bot in #11459
- chore: bump inference defaults from unsloth by @github-actions[bot] in #11270
- fix(gallery): align Higgs Audio checksum test by @localai-org-maint-bot in #11476
- feat(gallery): add Nemotron 3.5 Lightning 30B by @localai-org-maint-bot in #11477
- feat(gallery): add TwIL-LM3 variants by @localai-org-maint-bot in #11478
- fix(gallery): repair DeepSeek V4 fallback by @localai-org-maint-bot in #11480
- feat(gallery): add Fara1.5 4B variants by @localai-org-maint-bot in #11479
- feat(gallery): add LFM2.5 VL 1.6B variants by @localai-org-maint-bot in #11490
- feat(gallery): add Qwen3.8 27B variants by @localai-org-maint-bot in #11519
- feat(gallery): add DeepSeek V4 Pro 0813 by @localai-org-maint-bot in #11533
- feat(gallery): add LFM2.5 230M variants by @localai-org-maint-bot in #11526
- feat(gallery): add vllm.cpp text-generation models by @localai-bot in #11511
- feat(gallery): add HunyuanOCR variants by @localai-org-maint-bot in #11540
- feat(gallery): add Tess 4 27B variants by @localai-org-maint-bot in #11547
- feat(gallery): add AREX Turbo variants by @localai-org-maint-bot in #11551
- feat(gallery): add OvisOCR2 variants by @localai-org-maint-bot in #11549
- feat(gallery): add Qwen3.8 9B variants by @localai-org-maint-bot in #11557
- feat(gallery): add Ornith 1.0 9B variants by @localai-org-maint-bot in #11552
- feat(gallery): add Qwen3.8 Ridge variant by @localai-org-maint-bot in #11573
- feat(gallery): add Qwen3.8 small variants by @localai-org-maint-bot in #11571
- feat(gallery): add MiniCPM5 1B Q8 variant by @localai-org-maint-bot in #11561
- chore(model gallery): 🤖 add 1 new models via gallery agent by @localai-bot in #11596
- feat(gallery): add North Mini Code variants by @localai-org-maint-bot in #11559
- feat(gallery): add UI-Mate 9B variants by @localai-org-maint-bot in #11584
- feat(gallery): add Ling 3.0 Flash variants by @localai-org-maint-bot in #11598
- feat(gallery): add Hy-MT2 1.8B variants by @localai-org-maint-bot in #11599
- feat(gallery): add Carbon genomics variants by @localai-org-maint-bot in #11594
- chore(model gallery): 🤖 add 1 new models via gallery agent by @localai-bot in #11605
- feat(gallery): add Ornith 1.5 9B variants by @localai-org-maint-bot in #11603
- feat(gallery): add Gemma 4 agentic variants by @localai-org-maint-bot in #11622
📖 Documentation and examples
- docs: ⬆️ update docs version mudler/LocalAI by @localai-bot in #11415
- docs: remove broken stars counter by @richiejp in #11448
- docs(adopters): add walcz.de by @walcz-de in #11581
- docs(ecosystem): add Walcz to contributors by @walcz-de in #11582
👒 Dependencies
- chore: ⬆️ Update ggml-org/whisper.cpp to
592feef04a1802b18cbeffd0fd0eb5d02570c2ecby @localai-bot in #11416 - chore(model-gallery): ⬆️ update checksum by @localai-bot in #11418
- chore: ⬆️ Update ikawrakow/ik_llama.cpp to
40dffce6857b4fe051f096379dc464764c718458by @localai-bot in #11403 - chore: ⬆️ Update CrispStrobe/CrispASR to
17a6cc99422bfafadf7161e96dd7294c89da9c36by @localai-bot in #11404 - chore: ⬆️ Update ikawrakow/ik_llama.cpp to
f2328aa0c19954d0ab31a3de60fbf50e47c2429fby @localai-bot in #11421 - chore(model-gallery): ⬆️ update checksum by @localai-bot in #11423
- chore: ⬆️ Update antirez/ds4 to
84cc882352757baf628a1776badf7cc54d584e28by @localai-bot in #11432 - chore: ⬆️ Update ikawrakow/ik_llama.cpp to
a7c81affa48c6800d63111bdb33469a01d062daaby @localai-bot in #11431 - chore(model-gallery): ⬆️ update checksum by @localai-bot in #11433
- chore: ⬆️ Update 0xShug0/audio.cpp to
748c5e28f6a7228b8f38ad7142ca97d29584544bby @localai-bot in #11401 - chore: ⬆️ Update CrispStrobe/CrispASR to
8898b0fee199b1765321a5d0d8314aee51a34583by @localai-bot in #11422 - chore(deps): bump dompurify from 3.4.12 to 3.4.13 in /core/http/react-ui in the npm_and_yarn group across 1 directory by @dependabot[bot] in #11425
- chore: ⬆️ Update ikawrakow/ik_llama.cpp to
5763a901de3f525b65ed5574fd57623905685755by @localai-bot in #11452 - chore: ⬆️ Update CrispStrobe/CrispASR to
8b91f4c5490b88ae6193673c35644d91d8695821by @localai-bot in #11450 - chore: ⬆️ Update 0xShug0/audio.cpp to
9d6e7b39236e0151ad28a70fab0d538b84ce8718by @localai-bot in #11451 - chore(model-gallery): ⬆️ update checksum by @localai-bot in #11453
- chore: ⬆️ Update ggml-org/llama.cpp to
030ebb558a5820b444a8f836ed5cdd46c9b4bd7aby @localai-bot in #11454 - chore: ⬆️ Update mudler/vllm.cpp to
24306364ab8beaed9197604a042a56aaccfde493by @localai-bot in #11385 - chore: ⬆️ Update ikawrakow/ik_llama.cpp to
26ceed9d4091a1696cf50e2ed87e5767d5811d81by @localai-bot in #11475 - chore(model-gallery): ⬆️ update checksum by @localai-bot in #11471
- chore: ⬆️ Update mudler/depth-anything.cpp to
54abd5c0abfd1f394e01cb3c38f2e3af4daedf85by @localai-bot in #11481 - chore: ⬆️ Update mudler/vllm.cpp to
9fd9e8f34408d5dd21d7f9385e96fc755708950bby @localai-bot in #11472 - chore: ⬆️ Update ikawrakow/ik_llama.cpp to
c46ffaa5665cfb2d6cf372c9a054dbab896e14feby @localai-bot in #11482 - chore: ⬆️ Update vllm-metal (darwin) to
v0.3.0.dev20260812005333by @localai-bot in #11467 - chore: ⬆️ Update vllm-project/vllm cu130 wheel to
0.27.1by @localai-bot in #11468 - chore: ⬆️ Update CrispStrobe/CrispASR to
ce521ee178867ceaa5fdc11803616578c8936c19by @localai-bot in #11470 - chore: ⬆️ Update leejet/stable-diffusion.cpp to
de298c225bed97c3f9026b73cd7b71e7879bd41bby @localai-bot in #11469 - chore(deps): bump backend/rust/kokoros/sources/Kokoros from
7089168to29e99adby @dependabot[bot] in #11496 - chore(deps): bump packaging from 26.2 to 26.3 in /backend/python/coqui by @dependabot[bot] in #11497
- chore: ⬆️ Update ggml-org/llama.cpp to
84e908c625fb60992b4cdef8180fb12fa9b4c4bfby @localai-bot in #11473 - chore(deps): update transformers requirement from >=5.14.1 to >=5.15.0 in /backend/python/transformers by @dependabot[bot] in #11500
- chore: ⬆️ Update vllm-metal (darwin) to
v0.3.0.dev20260813121949by @localai-bot in #11507 - chore: ⬆️ Update ikawrakow/ik_llama.cpp to
981e5ea0d7579b4803c86afbb09a7cd7d7bf3bb8by @localai-bot in #11509 - chore: ⬆️ Update CrispStrobe/CrispASR to
cb082743c456ac77aec0947de36e6420a933da04by @localai-bot in #11510 - chore: ⬆️ Update ikawrakow/ik_llama.cpp to
43afea46c25a12aae6db1e3105643267164898b4by @localai-bot in #11527 - chore(model-gallery): ⬆️ update checksum by @localai-bot in #11525
- chore: ⬆️ Update vllm-metal (darwin) to
v0.3.0.dev20260814013332by @localai-bot in #11522 - chore: ⬆️ Update CrispStrobe/CrispASR to
cc498701f1a68d88dd489803ebad10053a924322by @localai-bot in #11523 - chore: ⬆️ Update ggml-org/whisper.cpp to
1fe009caeda75f69bc864d6370b10674e45a92bdby @localai-bot in #11524 - chore(deps): bump sentence-transformers from 5.6.1 to 5.7.0 in /backend/python/transformers by @dependabot[bot] in #11499
- chore(deps): bump vllm from 0.25.1 to 0.26.0 in /backend/python/vllm by @dependabot[bot] in #11402
- chore: ⬆️ Update ikawrakow/ik_llama.cpp to
8337e4cd3861406fc04e0854b1409cd1b027fbc9by @localai-bot in #11542 - chore: ⬆️ Update vllm-metal (darwin) to
v0.3.0.dev20260815085651by @localai-bot in #11541 - chore(deps): bump golang.org/x/net to v0.55.0 by @localai-org-maint-bot in #11544
- chore: ⬆️ Update mudler/vllm.cpp to
4880c5715f36445a30bd39d3349a06dc96085a11by @localai-bot in #11515 - chore: ⬆️ Update vllm-metal (darwin) to
v0.3.0.dev20260816085229by @localai-bot in #11553 - chore: ⬆️ Update CrispStrobe/CrispASR to
3721d402f7bcc911dd4143a58e3da1cc67f09cc2by @localai-bot in #11554 - chore: ⬆️ Update vllm-metal (darwin) to
v0.3.0.dev20260817081527by @localai-bot in #11570 - chore: ⬆️ Update CrispStrobe/CrispASR to
007fda47128aaba7e2ce2279b0947a1c13800d68by @localai-bot in #11569 - chore: ⬆️ Update ggml-org/llama.cpp to
0021a77de0a8966059dc94548fb3b96654e0bb12by @localai-bot in #11508 - chore: ⬆️ Update mudler/vllm.cpp to
76f2a6d84e41cc9a593146a9a958f63645542731by @localai-bot in #11555 - chore: ⬆️ Update 0xShug0/audio.cpp to
3407559d7c83b543bd60ca8fa2b630374e71083fby @localai-bot in #11474 - chore: ⬆️ Update vllm-metal (darwin) to
v0.3.0.dev20260818075955by @localai-bot in #11586 - chore: ⬆️ Update 0xShug0/audio.cpp to
92816fc5c48d33a58171b49678ccf884d70142fbby @localai-bot in #11587 - chore: ⬆️ Update ggml-org/whisper.cpp to
4834a2327d008ace3ec5a9ed00f51454bcabbc1cby @localai-bot in #11588 - chore: ⬆️ Update mudler/vllm.cpp to
6e99e4540b8d0811542a9d7690dc9aa0bda5812fby @localai-bot in #11589 - chore: ⬆️ Update ggml-org/llama.cpp to
60addddf3c567c43ec3caf70fc953fba3572d96fby @localai-bot in #11590 - chore: ⬆️ Update CrispStrobe/CrispASR to
a117d433855ecc36b0cb20bd1abddd852902ed13by @localai-bot in #11591 - chore: ⬆️ Update mudler/parakeet.cpp to
e75de9b6b9b688fd293aa22f7e27aa724ea286f8by @localai-bot in #11592 - chore: ⬆️ Update mudler/vllm.cpp to
438305e1577768ec0f75729456a4c8b9f425e2eeby @localai-bot in #11612 - chore: ⬆️ Update 0xShug0/audio.cpp to
a61da671b6a81c79071500954eea3c91c1a383ddby @localai-bot in #11613 - chore: ⬆️ Update leejet/stable-diffusion.cpp to
97d2990807fe6d558e395f8764198d7c7e7b411cby @localai-bot in #11616 - chore(model-gallery): ⬆️ update checksum by @localai-bot in #11615
- chore: ⬆️ Update CrispStrobe/CrispASR to
a153b09b37c90cd55cd9336fccbdf3ba7a289596by @localai-bot in #11611 - chore: ⬆️ Update NVIDIA/NeMo-Speech.cpp to
4f9676226f667d14608487df744f375db87127f8by @localai-bot in #11417
Other Changes
- fix(ci): parse current vllm-metal version pins by @localai-org-maint-bot in #11457
- fix(ci): refresh site counters through a PR by @localai-org-maint-bot in #11572
- feat(swagger): update swagger by @localai-bot in #11585
- fix(ci): test stale chunks in split bundle by @localai-org-maint-bot in #11595
Full Changelog: v4.8.2...v4.9.0
@ Every self-hosted release, weekly
One Monday email with the week's stable releases across the directory.
>_ Models LocalAI runs
Popular open-weight models with LocalAI support, with VRAM needs and licence at a glance.
More in AI
local-ai-toolkitSelf-hosted AI toolkit bundling an LLM proxy, visual workflow builder, and chat UI backed by PostgreSQL. Deploy with Docker Compose or Helm charts.
localmodSelf-hosted, open-source content moderation API for text and image safety with PII, toxicity, prompt injection, spam, and NSFW detection — runs 100% offline.
LocalRecallSelf-hosted, open-source RESTful API for managing knowledge bases and vector databases, designed for AI agents and chatbots with a web UI.
locate-anythingSelf-hosted web UI for NVIDIA LocateAnything-3B: find objects, text, and UI elements in images via plain-language prompts on your GPU.
LumabriLumabri is a self-hosted, open-source system for running LLMs across LAN computers by distributing layers with donor approval, offering terminal and web chat.
LynkrSelf-hosted, open-source LLM gateway for AI coding assistants that compresses tool tokens, caches prompts semantically, and routes by complexity across 14+ providers.
marimomarimo is an open-source, self-hosted reactive Python notebook replacing Jupyter and Streamlit. Stores notebooks as pure Python files with SQL, interactive UI, AI.
MarmotSelf-hosted, open-source data catalog and context layer: index tables, topics, queues and APIs with lineage, owners and a glossary, and serve it to AI agents.
matrixhubSelf-hosted, Hugging Face-compatible model registry that accelerates vLLM and SGLang inference with on-demand caching, RBAC, and multi-region replication.
mcp-context-forgeAn AI Gateway, registry, and proxy that sits in front of any MCP, A2A, or REST/gRPC APIs, exposing a unified endpoint with centralized discovery, guardrails and management. Optimizes Agent & Tool calling, and supports plugins.
mcp-gatewaySelf-hosted MCP gateway converting REST APIs into secured MCP endpoints with multi-tenant isolation, API key/OAuth auth, tool policies, and audit logging.
mcp-server-managerMCP Server Manager is a self-hosted, open-source tool that simplifies managing Model Context Protocol servers for AI clients via a web UI and single YAML config.


0 comments