Self-host vLLM
v0.33.0rc3High-throughput LLM inference and serving engine
- Stars
- 97 022
- Port
- 8000 (OpenAI-compatible API server)
- Requires
- Python
vllm/vllm-openai-rocmWhat is vLLM?
vLLM is a high-performance library for running large language model inference and serving, created at UC Berkeley's Sky Computing Lab. It supports over 200 model architectures from Hugging Face including decoder-only LLMs, mixture-of-experts, multi-modal, embedding, and reward models. Core capabilities include PagedAttention for efficient KV cache management, continuous batching, speculative decoding, multiple parallelism strategies (tensor, pipeline, data, expert), quantization formats like FP8, INT4, GPTQ, AWQ, and GGUF, and an OpenAI-compatible API server. The engine runs on NVIDIA, AMD, and Intel GPUs, Google TPUs, Apple Silicon, and CPUs. Installation options include pip, uv, and Docker. The project uses the Apache-2.0 license and has over 2000 contributors.
vLLM alternatives
LangfuseLangfuse is a self-hosted, open-source LLM engineering platform for tracing, prompt management, evaluations, and analytics — used by 50,000+ companies.CompareLangfuse with vLLM
mistral.rsmistral.rs is a self-hosted, open-source LLM inference engine in Rust serving OpenAI-compatible and Anthropic-compatible APIs with built-in web UI.Comparemistral.rs with vLLM
ggrunggrun is a self-hosted terminal UI for discovering, downloading, and running local GGUF models with automatic multi-GPU memory placement and an OpenAI-compatible API.Compareggrun with vLLM
Atomic ChatSelf-hosted desktop/mobile app for running LLMs locally with OpenAI-compatible API. 1000+ models, multiple engines, MCP, agent launching.CompareAtomic Chat with vLLM
llmfitOpen-source hardware-fit tool: a CLI, TUI and web dashboard that detects your CPU, RAM and GPU, then ranks hundreds of local LLM models, self-hosted with Docker.Comparellmfit with vLLM
gpustackOpen-source, self-hosted GPU cluster manager that deploys LLMs on NVIDIA, AMD or Ascend hardware via vLLM, SGLang and TensorRT-LLM behind OpenAI-compatible APIs.Comparegpustack with vLLM
ShimmyA self-hosted, open-source LLM inference server: a single Rust binary that serves GGUF models through an OpenAI-compatible API, as an alternative to Ollama.CompareShimmy with vLLM
tabbyAPITabbyAPI is a self-hosted, open-source FastAPI server providing an OpenAI-compatible REST API for running ExllamaV3-quantized LLMs on your own GPU hardware.ComparetabbyAPI with vLLM
Should you self-host vLLM?
vLLM scores 90/100 on selfhost.directory (how scoring works) and ranks #3 of 25 in LLM Inference. It has 97 022 GitHub stars, +3 352 in the last 30 days.
It shipped 14 stable releases in the past year. The latest, v0.32.0, came out on 17 October 2026. New versions land roughly every 2 weeks.
Setup is moderate: expect some configuration beyond a single command. An official Docker image is available. It is Apache-2.0-licensed and written mostly in Python.
Its closest alternative on selfhost.directory is Ollama (94/100, 187 182 stars). It scores higher overall, so compare the two before you pick. See vLLM vs Ollama side by side.
Features
- Continuous Batching
- Disaggregated prefill/decode/encode
- Distributed Inference
- Embedding and retrieval models
- Multi-LoRA Support
- Multi-modal model support
- OpenAI Compatible API
- PagedAttention
- Quantization Support
- Reward and classification models
- Speculative Decoding
- Streaming Outputs
- Structured output generation
- Tool calling and reasoning parsers
vLLM README
| Documentation | Blog | Paper | Twitter/X | User Forum | Developer Slack |
🔥 We have built a vLLM website to help you get started with vLLM. Please visit vllm.ai to learn more. For events, please visit vllm.ai/events to join us.
vLLM is a fast and easy-to-use library for LLM inference and serving.
Originally developed in the Sky Computing Lab at UC Berkeley, vLLM has grown into one of the most active open-source AI projects built and maintained by a diverse community of many dozens of academic institutions and companies from over 2000 contributors.
vLLM is fast with:
- State-of-the-art serving throughput
- Efficient management of attention key and value memory with PagedAttention
- Continuous batching of incoming requests, chunked prefill, prefix caching
- Fast and flexible model execution with piecewise and full CUDA/HIP graphs
- Quantization: FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, GGUF, compressed-tensors, ModelOpt, TorchAO, and more
- Optimized attention kernels including FlashAttention, FlashInfer, TRTLLM-GEN, FlashMLA, and Triton
- Optimized GEMM/MoE kernels for various precisions using CUTLASS, TRTLLM-GEN, CuTeDSL
- Speculative decoding including n-gram, suffix, EAGLE, DFlash
- Automatic kernel generation and graph-level transformations using torch.compile
- Disaggregated prefill, decode, and encode
vLLM is flexible and easy to use with:
- Seamless integration with popular Hugging Face models
- High-throughput serving with various decoding algorithms, including parallel sampling, beam search, and more
- Tensor, pipeline, data, expert, and context parallelism for distributed inference
- Streaming outputs
- Generation of structured outputs using xgrammar or guidance
- Tool calling and reasoning parsers
- OpenAI-compatible API server, plus Anthropic Messages API and gRPC support
- Efficient multi-LoRA support for dense and MoE layers
- Support for NVIDIA GPUs, AMD GPUs, Intel GPUs, and x86/ARM/PowerPC CPUs. Additionally, diverse hardware plugins such as Google TPUs, Intel Gaudi, IBM Spyre, Huawei Ascend, Rebellions NPU, Apple Silicon, MetaX GPU, Iluvatar CoreX GPU, and more.
vLLM seamlessly supports 200+ model architectures on Hugging Face, including:
- Decoder-only LLMs (e.g., Llama, Qwen, Gemma)
- Mixture-of-Expert LLMs (e.g., Mixtral, DeepSeek-V3, Qwen-MoE, GPT-OSS)
- Hybrid attention and state-space models (e.g., Mamba, Qwen3.5)
- Multi-modal models (e.g., LLaVA, Qwen-VL, Pixtral)
- Embedding and retrieval models (e.g., E5-Mistral, GTE, ColBERT)
- Reward and classification models (e.g., Qwen-Math)
Find the full list of supported models here.
Install vLLM with uv (recommended) or pip:
uv pip install vllm
vllm download-kernels # precompiled kernels for faster startupOr build from source for development.
Visit our documentation to learn more.
We welcome and value any contributions and collaborations. Please check out Contributing to vLLM for how to get involved.
If you use vLLM for your research, please cite our paper:
@inproceedings{kwon2023efficient, title={Efficient Memory Management for Large Language Model Serving with PagedAttention}, author={Woosuk Kwon and Zhuohan Li and Siyuan Zhuang and Ying Sheng and Lianmin Zheng and Cody Hao Yu and Joseph E. Gonzalez and Hao Zhang and Ion Stoica}, booktitle={Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles}, year={2023} }
- For technical questions and feature requests, please use GitHub Issues
- For discussing with fellow users, please use the vLLM Forum
- For coordinating contributions and development, please use Slack
- For security disclosures, please use GitHub's Security Advisories feature
- For collaborations and partnerships, please contact us at collaboration@vllm.ai
- If you wish to use vLLM's logo, please refer to our media kit repo
How to install vLLM
Install vLLM with Python
uv pip install vllm vllm download-kernels # precompiled kernels for faster startup
Requires Python 3.10+, Python 3.12+ recommended
vLLM releases & version history
v0.32.0
Highlights
This release features 814 commits from 363 contributors (121 new)!
- Structured decisions and generic parsers: the new
POST /v1/systemoneendpoint answerschoice,noulandscorequestions over a structured state (#59299), is served by default (#60677, #60696), and also runs onCloudflare/clef-omni(#61064),LiquidAI/d1-3B(#60676) and DiffusionGemma (#60466). The newhftool and reasoning parser (--tool-call-parser hf,--reasoning-parser hf) reads the checkpoint'sresponse_template, so models that ship it need no model-specific parser (#58604, #58602), in both the Python and Rust frontends (#59005). - Low-precision KV cache everywhere: NVFP4 KV cache now runs on SM8x and SM12x (#46963), SM90 (#60620) and with XQA decode on SM12x (#60452), and on MiniMax-M3 sparse attention on SM100 (#59300); GLM-5.x defaults to an FP8 KV cache on SM100, halving KV memory per rank with 2.3-5.5% more throughput at a 7-11% TTFT cost in the PR's benchmark (#60140, #60281);
fp8_ds_mlaon SM90 gives GLM-5.3-Flash up to 31% more output throughput (#59246); ROCm gains the 4-bitultraquant_4bitKV cache with about 2x the density of FP8 (#57057). - Model Runner V2: a GPU n-gram speculator with variable-length drafts (#40704), Elastic EP (#53934), pipeline parallelism with prefill context parallelism for MLA models (#59139), a stock torch.compile mode (#59079), and MRV2 is now the default on ROCm with the model-specific MRV1 exceptions removed (#56360).
- Faster startup and serving:
vllm download-kernelsinstalls FlashInfer's precompiled kernels (#58765);vllm preload --preload-autotunekeeps the FlashInfer autotune table warm across restarts (#60085); a shared-memory tensor arena broadcasts large multimodal tensors to workers with one copy (#51207); tokenizers and processors load from the local snapshot (#60825); and with--api-server-countabove 1 each connection goes to the least-loaded API worker, for 43-85% more req/s on short requests (Qwen3-0.6B, 32 input / 1 output tokens) in the PR's benchmark (#60723). - Model support: new EmbeddingGemma 2 omni-modal embeddings (#60254), MiniCPM-V 4.7 (#58674) and Nemotron 3.5 ASR (#59827); video input in the Transformers modeling backend (#57441); GLM-5.3 sparse MLA index conversions reused across layers (3.5-3.9x on that kernel, #59464) and fused multi-step draft decode (13.3% at concurrency 1, #57443); HiSparse per-request residency caching for up to 58% more throughput on GLM-5.3-NVFP4 on B200 with long, fully GPU-resident contexts at high concurrency (#60083).
- Security: fixes for GHSA-4hhp-h66f-j5j7 (chat template resource exhaustion, #50300), GHSA-q97c-p368-75c4 (tokenizer
max_token_id, #59491) and GHSA-frvg-qw88-hww9 (P2P ZMQ socket exhaustion, #51504); CORS no longer allows every origin by default (#61009); untrusted images are limited to raster formats (#60022); shared-memory multimodal cache handles are signed (#59357); and JSON schemas nested deeper than 64 levels are rejected (#60036). - Breaking changes: Python 3.10 dropped (#60402); default CORS origins restricted (#61009);
transformers>=5.16.1,<5.20.0andtokenizers>=0.23.2(#57387, #59796); torch.compile sequence parallelism and AsyncTP passes removed (#60517); cascade attention removed (#61126);mamba_cache_mode="all"removed (#58997);max_tokensno longer reserved from the context window (#57035); Responses API reasoning stream events renamed (#59652).
Release Artifacts
Python Wheels
| Platform | Install |
|---|---|
| PyPI (CUDA 13.0) | pip install vllm |
| PyPI (CUDA 13.0, uv) | uv pip install vllm --torch-backend=auto |
| ROCm | pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.32.0/rocm723 |
| XPU | uv pip install vllm --extra-index-url https://wheels.vllm.ai/0.32.0/xpu --extra-index-url https://download.pytorch.org/whl/xpu --index-strategy unsafe-best-match |
Docker Images
| Platform | Docker Image |
|---|---|
| CUDA 13.0 (Default) | docker pull vllm/vllm-openai:v0.32.0 |
| CUDA 12.9 | docker pull vllm/vllm-openai:v0.32.0-cu129 |
| CUDA 13.0 + Ubuntu 24.04 | docker pull vllm/vllm-openai:v0.32.0-ubuntu2404 |
| CUDA 12.9 + Ubuntu 24.04 | docker pull vllm/vllm-openai:v0.32.0-cu129-ubuntu2404 |
| ROCm | docker pull vllm/vllm-openai-rocm:v0.32.0 |
| CPU | docker pull vllm/vllm-openai-cpu:v0.32.0 |
| XPU | docker pull vllm/vllm-openai-xpu:v0.32.0 |
Other Artifacts
Pre-built release artifacts are available in the Assets section at the bottom of this page, including:
- Source distribution tarball
- CUDA 12.9 Python wheels for x86_64 and arm64
- CUDA 13.0 Python wheels for x86_64 and arm64
- CPU Python wheels for x86_64, arm64, and macOS
- XPU Python wheel for x86_64
Model Support
- New models: EmbeddingGemma 2 (
google/embeddinggemma-2), an omni-modal text/image/audio/video embedding model, with unused modality encoders skipped through--limit-mm-per-prompt(#60254, #60289); MiniCPM-V 4.7 (#58674); Nemotron 3.5 ASR (nvidia/nemotron-3.5-asr-streaming-0.6b) complete-audio transcription; native realtime streaming is not yet supported (#59827);Cloudflare/clef-omni(#61064) andLiquidAI/d1-3B(#60676) on/v1/systemone; PLaMo 3 reasoning and tool parsers (#56575); and the zerank-2 reranker score template (#60415). - Transformers modeling backend: video inputs (#57441), and Glm, Arcee, CWM, Mellum (#59679), GPT-NeoX, Phi, Seed-OSS and Jais2 (#59701) now served through it.
- New model capabilities: LoRA for BERT, RoBERTa and XLM-RoBERTa embedding models (#61033); Granite as an EAGLE3/DFlash target (#60855); Qwen3-ASR EAGLE3 (#52824); Sarvam MLA EAGLE3/DSpark with PP (#55902); GLM-5.3-Flash with the DFlash2 drafter (#56983); and MiniMax-M3 EAGLE3 at PP > 1 on MRV2 (#57197) with MTP sharing target embeddings under PP (#58648).
- GLM-5.3 and GLM-5.3-Flash: sparse MLA index conversion reused across layers (#59464), fused multi-step draft decode (#57443), FP8 KV cache default on SM100 (#60140, #60281),
fp8_ds_mlaon SM90 (#59246), qlnorm skipped for MHA (4.4-7.7% TTFT, #58845), long-context indexer prefill sharded across TP (up to 25% lower TTFT at 1M context, #54951), DCP for the kpool indexer (#59211, #60032), KDA prefill checkpoints with FlashKDA (#56960), FP8 KV on SM90 sparse MLA (#55222),--mamba-ssm-cache-dtypehonored for KDA state (#59147), a generic packed KV layout (#57169), MTP index sharing restored (#61079), and fixes for vision-tower crashes (#59126), non-square image encoder budgets (#59565), DP startup (#56974), video timestamps (#55647), DFlash2/EAGLE3 aux hidden states (#60763) and null-block writes (#59528). - DeepSeek-V4 and V4.1: decoder-side SWA bounded replay (#58132) with replay layers in CUDA graphs and trimmed in PIECEWISE graphs (
compilation_config.decoder_replay_trim_threshold, #59532) plus last KV-source layer trimming (+3.6%, #59894), packed FP8 Engram exchange across DP (#59923), faster CPU-offloaded Engram lookups (#59327), fused DeepSeek V4 + MTP drafting (#58463), encoder CUDA graph packing for V4.1 and Kimi-K3 (11% lower mean TTFT, #58925), EPLB for DSpark drafts (#57549), and fixes for NVFP4 checkpoints with 0% draft acceptance (#59655), MegaMoE shared-expert finalize ordering (#59927), worst-case vision dummy inputs (#57898, #59271), compressor ring null-block writes (#58560), Triton 3.9 indexer quantization (#60716) and decoder-replay CUDA graphs with FlashInfer EP MoE backends (#61041). - Qwen: Qwen3.8-Flash-Next (Qwen4Exp) merged QSA projections in one GEMM (1.45-1.68x, #59533), fused QSA pre-indexer (#57097), sequence parallelism through
--enable-hc-sp(#56322), SM100 low-latency decode GEMM plans (up to 2.7x vs cuBLAS on small-M, #59214),--kv-cache-dtype-skip-layershonored (#60023), and PLE loading for INC (#59990), Quark (#59443) and pre-SM89 GPUs (#60142); Qwen3-VL/Qwen3.5/Qwen3.8 M-RoPE cache 4x smaller and Qwen3.5 YaRN matching Transformers (#59853); Qwen3-Omni M-RoPE fixes (#58890, #59842); Qwen3-ASR on the upstream processor (#57146); Qwen3-VL unknown video fps (#60501) and per-video options with the multimodal cache (#55203). - Kimi-K3 and MiniMax-M3: Kimi-K3 FlashInfer fused speculative KDA on SM100 (#54255), fused vision processor (10-20x preprocessing at 1080p-8K, #59278), fused K/V pack on the DCP prefill path (#56881), and DSpark KV cache back to 4 groups (#59733); MiniMax-M3 MSA prefill with an E4M3 query (26-37% faster with FP8 KV, #60684), FP8 indexer cache on non-SM100 GPUs (#59081), Triton indexer tuning (#56151, #59481), and indexer score fixes (#60194, #54597).
- Multimodal: device-side input normalization for Qwen3-VL/Qwen3.5/Qwen4Exp (#51289), GLM-V and GLM-OCR (#55389), Llama Nemotron VL (#57928) and Kimi K2.5/K3 (#59278), kept fused under
compile_mm_encoder(#59195); encoder inputs sorted by modality (6.5-8.2% more req/s, #60196); Mistral 3 preprocessing 8% faster (#57531); Pixtralcompile_mm_encoder(#57168); and fixes for Llama 4 / Mistral Large 3 EAGLE drafters (regression since v0.28.0, #57633), Phi-4 MM audio OOM (#60614), DeepSeek-OCR zero-sum pixels (#59417), MiMo-V2.5/V2.6 with fastsafetensors (#58142), MiMo-V2.6 FP8 KV cache (#58128), Voxtral realtime (#45022), Nemotron Super vision LayerNorm (#61401), K-EXAONE config and MTP RoPE (#60187), Nemotron Parse tiedlm_head(#60272), LongCat-Flash MTP loading (#60080) and Ling-3.0-flash-VL (#59026).
Engine Core
- Model Runner V2: GPU n-gram speculator (#40704), Elastic EP with drain-then-commit scaling (#53934), PP with PCP for MLA models (#59139),
STOCK_TORCH_COMPILEmode (#59079), dynamic K in autoregressive speculators (#57053), MTP sampling-mask replay (#59359) and per-module LM heads (#58921), non-blocking H2D copies on WSL2 (#60215), and fixes for int64 index offsets at long context (#60736), warmup (#61013), GDN prefill checkpoints (#59536) and MRV2+PP+MTP align state (#60210). - Speculative decoding: fused multi-step draft decode on FlashInfer trtllm-gen (#58371), FlashInfer ReplaySSM for Mamba2 MTP (#52928), in-graph metadata for DFlash/DSpark drafts (#60692), and correctness fixes for ngram on hybrid recurrent models (#56531), Mamba prefill checkpoints with MTP and prefix caching (#59759, #60797), stop-string trimming (#47616), draft model config (#61240), adaptive block verification (#60945), DSpark bonus slots (#59105) and draft models on the Transformers backend (#59293).
- Hybrid models and prefix caching: Mamba2 internal prefill checkpoints for NemotronH, Granite-4.0-H, Falcon-H1 and Zamba2 (#57329), stale block hashes dropped when streaming sessions truncate (#59103), and fixes for quantized Mamba-2 with TP (#58083), spec decode page sizes (#59975), prefix-cache bounds (#55601, #60533) and sm90 Mamba2 NaNs (#61718).
- Sleep and RL: sleep releases workspace scratch (up to 4.5 GiB/GPU, #59156) and KV-init state (about 4.6 GiB/GPU, #59158), opt-in
sleep_mode_offload_cudagraph(#59160, #59523); token-level weight-version spans on outputs (#59582, #61070); weight-transfer communicators released without stopping inference (#61060);--sleep-preserve-parameter-names 'draft.*'(#61046);modelexpressweight-transfer backend (#58399);POST /weight_checker(#51350); weight-transfer metrics (#55781);--enable-nccl-comm-suspendreleasing the MNNVL workspace (#59360); andsleep(level=2)keeping FP8 KV scales (#57163). - HiSparse: per-request residency cache (up to 58% at high concurrency with GPU-resident contexts, #60083), lower scheduler overhead (#57930), GPU residency owned by HiSparse (#60647), less staging memory (#60429), host cache gauges (#58949, #59485), and fixes for decode token limits (#60682) and TP hangs (#60941).
- Startup and memory: shared-memory tensor arena (
--enable-shm-tensor-arena, on by default, #51207), Fast Start with PP (#55477), daemon-held weights counted againstgpu_memory_utilization(#57298), autotune preload (#60085), local tokenizer snapshots (#60825),sparse_mla_prefill_workspace_factorsaving several GiB at 1M context (#61023), cached memory reclaimed during weight loading (#57516), fewer Triton sampler warmup compiles (#58605), sharded state loader with PP (#52110), andreload_weightswith Run:ai streamer (#58124). - Kernels and runtime: MoE weighted-sum tuning (1.17-2.37x, #59731), flash-maxsim kernels for ColBERT/ColPali scoring (#40337), fallback detokenization with 69% less CPU at 32K tokens (#60497), faster KV block free queue (#60269), Triton kernels off
tl.make_block_ptr(#60502), fusion passes restored under PyTorch 2.15 (#57299) and static FP8 quant fusion on recent torch (#61108). - Batch invariance: custom all-reduce under
VLLM_BATCH_INVARIANT=1(#58623), deterministic split-K LoRA shrink (#59377), every attention backend checked (#60271), SM120 TP configs (#58495), and ROCm support (#52231). - Observability:
vllm:prompt_tokens_cached_by_source_total(#56318),vllm:num_requests_kv_fetch_by_stage(#58874),vllm:admission_rejections_total(#55812),vllm:watermark_requests(#58715),--custom-histogram-buckets(#48867), a JSON log formatter viaLoggingConfig.formatter(#58739), and per-session profiling controls (#57875).
Large Scale Serving
- MoE communication: FlashInfer CuteDSL MegaMoE for more models via
--moe-backend flashinfer_cutedsl(#54049), FP8 combine for the FlashInfer one-sided all2all (+4% at DEP8, #57995), NCCL symmetric-memory MoE reduce-scatter (9.6-16% lower latency, #49194), DeepEPv2 hybrid NVLink+RDMA mode on by default (#57991), Kimi-K3 shared experts with DeepEP v2 (#56875), and opt-in EPLB migration batching (#52641). - Context parallelism: decode requests sharded across PCP ranks (#52162) and TokenSpeed MLA with block-interleaved DCP for heterogeneous NIXL disaggregation (#59462).
- KV connectors: NIXL host-buffer copies coalesced (#54483), mixed block sizes (#58860), peer metadata recovery (#55471) and Qwen3.8-Flash-Next MTP acceptance in P/D (#59997); Mooncake coalesced hybrid/MLA RDMA regions (#57952) and bootstrap retries (#55923); MoRIIO Kimi-K3 DSpark READ (#57700); DeepSeek-V4.1 P/D skipping bounded replay for loaded KV (#59197); and KVCR pool-and-index API (#59899).
- Data parallelism: streaming requests pinned to their engine (#59017), per-engine RNG streams (#59788), and placement groups without
ray[default](#57648).
Hardware & Performance
- NVIDIA: native 64-head FlashMLA sparse prefill on SM100 (1.16-1.18x TTFT, #60029), block-FP8 DeepGEMM experts skipping padding (#59128), SM120 split-K attention (#58482), DGX Spark Qwen4Exp GEMM plans (#59753, #59632, #60027), OAI Triton MXFP4 MoE on SM12x (#58877), and FlashInfer autotune cache per rank (#57635).
- AMD ROCm: DeepSeek-V4 AITER MegaMoEV2 via
--moe-backend aiter_mega_moe(#59685) with opt-in shared-expert fusion (3-8%, #61004), opt-in mono decode for DeepSeek-V4.1-Flash on MI355X (VLLM_ROCM_MONO_DECODE=1, #60397), opt-in Gluon sparse MLA (up to +16% on GLM-5.2, #53492), paged MXFP4 indexer (#58671), opt-in a4w4 MoE for DSv4 (#58819),ultraquant_4bitKV cache (#57057),aiter_flydslGDN prefill (#57560), Kimi-K3 prefill checkpoints (about 20% better TTFT, #58344), fused MLA decode (#57640) and per-rank latent MoE shards (#59591), GLM-5.3-Flash split-K sparse MLA decode (5-15% lower ITL, #58584), nativemerge_attn_states(2.4x, #60159),ROCM_SEGMENTED_ATTNfor RDNA (#59132), gfx11 W4A16 tuning (up to 23% TTFT, #56301), MRV2 default with model-specific MRV1 exceptions removed (#56360), and fixes for sleep-mode segfaults on ROCm 7.2+ (#60367) and MLA weight reloads (#60389). - Intel XPU: MLA in
FLASH_ATTN_MLA(#58297), Arc Pro B70 FP8 GEMM (2.0-2.73x, #56063) and fused MoE configs (#53065), fused SYCL LayerNorm (#57172), and fixes for Gemma-2 softcap (#58936) and DP+EP MoE (#57787). - CPU: vectorized sampler (up to 58x on s390x, #53913), aarch64 causal conv1d (about 9%, #54093), POWER10 AWQ/GPTQ (#59149), and PowerPC
dtype=autochoosing bfloat16 (#58528). - Other: tpu-inference v0.31.0 (#60190) and out-of-tree platforms registering multiple attention backends (#41091).
Quantization
- New capabilities: online re-quantization of linear layers (
--quantization-config.linear fp8_per_channel, #55684), per-token NVFP4 CuTe-DSL MoE (#50030) with ReLU2 (#56740), MXFP4 MoE oracle with CUTLASS W4A4 and BF16 fallback (#60079), compressed-tensors MXFP4 MoE with FP8 activations (#59815), ModelOpt IQ2_XS/IQ2_XXS/Q8_0 on SM120/SM121 (#59604),int4_per_token_headKV with non-power-of-two heads (#56198), and Humming preferred over Marlin on SM90 (#56997). - Fixes: silent CUTLASS NVFP4/MXFP4 MoE corruption under EP (#59042), GPTQ with non-default safetensors names (#60411),
moe_wna168-bit asymmetric zero points (#58950), and ModelOpt mixed-precision NVFP4 detection (#56050).
API & Frontend
- New endpoints and options:
/v1/systemone(#59299, #60677, #60696),hfparser (#58604),vllm download-kernels(#58765),output_mode: "text"andstop_reasonin/inference/v1/generate(#58588, #60251),mm_payloadfor render requests (#61427, #60728), Responsesmin_pandstop_token_ids(#48084, #56592),LLM.chat(priority=...)(#57243), and/generative_scoringcache_salt(#60841). - Behavior:
max_tokensis an upper bound instead of a reservation, so long prompts are no longer rejected (#57035); least-loaded API worker selection (#60723); tool parser compatibility checked at startup (#59749). - Tool calling and structured output:
parallel_tool_calls=falseenforced (#50502, #59298),tool_choice="required"streaming (#60354) and native XML parsers (#47512), Step-3.5 andminimax_m3on the parser engine (#59321, #59743), xgrammarpropertyNames(#57594), and LoRA PEFTrank_pattern/alpha_patternapplied with unsupported features rejected (#59801, #60225). - Rust frontend:
--grpc-port(#59659),--stream-interval(#60589),logprob_token_ids(#59549),Control.Shutdownwithvllm-proto0.5.0 (#59316),qwen3_jsonparser (#60136), and Kimi K3 grammar fixes (#59190, #59395).
Security
- GHSA-4hhp-h66f-j5j7: chat templates are validated statically and rendered under a timeout (
VLLM_CHAT_TEMPLATE_RENDER_TIMEOUT), user-suppliedchat_templatein Responseschat_template_kwargsis rejected, and the Rust frontend caps template execution (#50300). - GHSA-q97c-p368-75c4: tokenizer
max_token_idoff-by-one fixed (#59491). - GHSA-frvg-qw88-hww9: P2P ZMQ peer sessions are bounded by
VLLM_P2P_MAX_PEERS(default 128),VLLM_P2P_HANDSHAKE_TIMEOUT_S(default 30) andVLLM_P2P_IDLE_TIMEOUT_S(default 300), so unique peer addresses can no longer exhaust ZMQ sockets (#51504). - Default CORS settings no longer allow every origin in the Python and Rust frontends; set
--allowed-originsto permit browser origins (#61009). - Untrusted images are opened with a raster-format allowlist (
ALLOWED_IMAGE_FORMATS), denying EPS/PostScript and WMF (#60022); shared-memory multimodal cache handles are HMAC-signed and forged handles rejected (#59357). - Structured-output JSON schemas nested more than 64 levels return 400 (#60036); chunked
/v1/embeddingsrequests keepcache_salt(#47696); Phi-4 MM audio no longer holds multi-GiB attention matrices (#60614); DeepSeek-OCR zero-sum pixel tensors no longer kill EngineCore (#59417). - Security dependency bumps:
aiohttp>=3.14.3,pillow>=12.3.0, cryptography, urllib3, pyjwt and others (#59249, #59315, #59427).
Dependencies
transformers>=5.16.1,<5.20.0(minimum raised from 5.10.4, #57387, #59762) andtokenizers>=0.23.2(#59796).- NIXL 1.5.0 (#56907), CuTe DSL 4.8.0 block-scale APIs (#59480),
humming-kernels==0.1.17with initial Rubin support (#61556, #61858), InstantTensor 0.2.2 (#61167), andflashinfer-jit-cacheexcluded from wheelinstall_requires(#60052). - ROCm: AITER 0.1.24.post1 (#59794) and MoRI v1.2.3.post1 (#56073). XPU: auto-round-lib 0.16.0 (#60976). TPU: tpu-inference v0.31.0 (#60190).
Build
- Python 3.11-3.14 supported (#60402); macOS wheels built with the stable ABI (#60312); CUDA source builds require GCC 13 and fail early otherwise (#60259); Docker images compile Python bytecode at build time for faster container startup (#55422);
VLLM_USE_PRECOMPILEDfinds a compatible published wheel within the last 20 commits (#58871); the vendored DeepGEMM usesTORCH_LIBRARY(abi3, #48962).
Breaking Changes & Deprecations
- Python 3.10 is no longer supported; the minimum is Python 3.11 (#60402).
- Default CORS origins changed from
*to none; set--allowed-originsfor browser clients (#61009). transformers>=5.16.1,<5.20.0andtokenizers>=0.23.2are required (#57387, #59796).- The torch.compile sequence parallelism and AsyncTP passes were removed, along with
PassConfig.enable_sp,fuse_gemm_commsandsp_min_token_num; model-level sequence parallelism is unaffected (#60517). - Cascade attention was removed;
--disable-cascade-attnstill parses but only warns and will be removed in v0.33.0 (#61126). mamba_cache_mode="all"was removed;--mamba-cache-mode allnow fails validation (#58997).- Previously deprecated APIs were removed:
wait_for_inflight_requests(usemode="wait"),AsyncLLM.pause_generation(),IOProcessor.output_to_response,vllm.scripts.main()andvllm bench throughput --dataset(#60923). Thesonnetbenchmark dataset was removed as scheduled (#60129). Tensorizer support was removed from LoRA (#60024), and the PyO3vllm._rust_tool_parserbridge was removed (#59744). - API changes:
max_tokensis treated as an upper bound on output rather than reserved from the context window (#57035); Responses API streaming usesresponse.content_part.added/donewithpart.type="reasoning_text"instead ofresponse.reasoning_part.added/done(#59652);SamplingParams.prompt_logprob_token_idsis per row (#56984);/inference/v1/generatelogprobs use integer token IDs (#58181); sleep API responses are standardized and invalid levels return 400 (#52864);return_mm_kwargsis deprecated in favor ofmm_payload(#61427). - Watermarking: when the engine has watermarking configured, an omitted request field means enabled and only
falseopts out; unsupported combinations warn and fall back (#56801). - Rust frontend:
qwen3_xmlnow parses Qwen3-Coder XML (alias ofqwen3_coder); use the newqwen3_jsonfor the JSON format (#60136). - New defaults: shared-memory tensor arena on (#51207); GLM-5.x FP8 KV cache on SM100 with
--kv-cache-dtype auto(#60140, #60281);VLLM_DEEPEP_V2_ALLOW_HYBRID_MODE=1(#57991); Humming preferred over Marlin on SM90 (#56997);sparse_mla_prefill_workspace_factor2 above 256Kmax_model_len(#61023); MRV2 on ROCm with model-specific MRV1 exceptions removed (#56360);VLLM_ROCM_AITER_MLA_DCP_VERIFY=auto(#59965);dtype=autopicks bfloat16 on PowerPC (#58528); MXFP4 W4A4 checkpoints fall back to W4A16 with a warning when no W4A4 kernel exists (#60079). - Stricter validation: incompatible
--prefix-match-unit(#58021),max_num_scheduled_tokens=0(#60714), HiSparse withcudagraph_mode=FULL(#59688), and negative--device-ids(#49053) are rejected at startup.
New Contributors
- @0xViking made their first contribution in #60053
- @1fanwang made their first contribution in #40986
- @a-sidorova made their first contribution in #55335
- @AbhayShuklaIIT made their first contribution in #59763
- @ahmed-bsod made their first contribution in #58655
- @ai-yang made their first contribution in #58088
- @AIwork4me made their first contribution in #54706
- @AlanFokCo made their first contribution in #57163
- @AMD-yanfeiwang made their first contribution in #61004
- @aniskumar-nv made their first contribution in #53020
- @Anurag-M1 made their first contribution in #57693
- @Aravind-11 made their first contribution in #60098
- @awesome-pro made their first contribution in #58895
- @Ayu190505 made their first contribution in #57995
- @beenpow made their first contribution in #57648
- @big-yellow-duck made their first contribution in #59132
- @blueberry808 made their first contribution in #58932
- @bluemelov1 made their first contribution in #58560
- @bnkaiteam made their first contribution in #60507
- @brandonfzhang made their first contribution in #56881
- @Btwestyo made their first contribution in #59924
- @bvolpato made their first contribution in #55203
- @C1ves made their first contribution in #58890
- @charlesli640 made their first contribution in #59977
- @cheese-cakee made their first contribution in #59079
- @CorgiBoyG made their first contribution in #55596
- @damienlaine made their first contribution in #45022
- @danichan-mkm made their first contribution in #59285
- @danigarciaoca made their first contribution in #56372
- @debanshd made their first contribution in #57476
- @devpramod made their first contribution in #60855
- @devshah-cohere made their first contribution in #60286
- @DGlan made their first contribution in #59875
- @dhairyashilRG made their first contribution in #58911
- @DOCCA0 made their first contribution in #52641
- @Dovis01 made their first contribution in #56994
- @ebfio made their first contribution in #58165
- @ejrtks1020 made their first contribution in #59180
- @eky-amd made their first contribution in #54857
- @gaby made their first contribution in #59126
- @haotiaa made their first contribution in #60766
- @harshaladhav-amd made their first contribution in #58268
- @hdq66666 made their first contribution in #59695
- @hoaivoke made their first contribution in #61033
- @hpdkhoa made their first contribution in #60747
- @hubert-marek made their first contribution in #61000
- @hubunt made their first contribution in #47512
- @huishi-hs made their first contribution in #58053
- @hyukjlee made their first contribution in #59965
- @Id545 made their first contribution in #59247
- @ieaves made their first contribution in #52824
- @ischencheng made their first contribution in #60763
- @Jeff-Tseng-dev made their first contribution in #59396
- @jiangyunfan1 made their first contribution in #57780
- @JimmyWang0417 made their first contribution in #61081
- @jin-amd made their first contribution in #58904
- @JRJRJPRO made their first contribution in #59879
- @Kevinli24 made their first contribution in #58878
- @Kylinny made their first contribution in #60354
- @Lcy0801 made their first contribution in #58975
- @Legend-2727 made their first contribution in #59293
- @lianakoleva made their first contribution in #60136
- @LiRunGuo made their first contribution in #56198
- @lobo235 made their first contribution in #60364
- @LostFox11 made their first contribution in #58542
- @MadhavJivrajani made their first contribution in #57635
- @MatthewCYM made their first contribution in #57146
- @Mazukiri made their first contribution in #60036
- @MohammadHijjawi97 made their first contribution in #59072
- @MohitAMD made their first contribution in #55053
- @nishantrevur made their first contribution in #59541
- @njriasan made their first contribution in #60502
- @NNNtrance made their first contribution in #55647
- @NolenLiang made their first contribution in #56974
- @notimesea made their first contribution in #59105
- @ntheanh201 made their first contribution in #57734
- @nv-hwoo made their first contribution in #58399
- @ojeda-e made their first contribution in #59107
- @Ostring24 made their first contribution in #50502
- @peperunas made their first contribution in #58938
- @pondzikk made their first contribution in #55601
- @praveingk made their first contribution in #60161
- @puneetsharma21 made their first contribution in #48074
- @rdlh made their first contribution in #60415
- @rmhaskarnvidia made their first contribution in #59081
- @roG0d made their first contribution in #55937
- @roipony made their first contribution in #40337
- @shijie-lyu made their first contribution in #55771
- @shiyuan680 made their first contribution in #51350
- @shreyanshj10 made their first contribution in #60359
- @siliangchen-amd made their first contribution in #57108
- @Sohaib-Ahmed21 made their first contribution in #59827
- @spa5k made their first contribution in #60497
- @sricursion made their first contribution in #60307
- @SunnyR made their first contribution in #59811
- @superdiaodiao made their first contribution in #55717
- @surajsharan made their first contribution in #60225
- @surawut-jirasaktavee made their first contribution in #60876
- @Thiago4532 made their first contribution in #58014
- @TommyTranX made their first contribution in #48039
- @TylerMa-debugg made their first contribution in #59917
- @ultramancode made their first contribution in #60786
- @usmanmateen made their first contribution in #59336
- @v-shobhit made their first contribution in #59655
- @vaclavcadek made their first contribution in #49602
- @viktorkou made their first contribution in #58124
- @vovani made their first contribution in #60194
- @WarrierRajeev made their first contribution in https://github.com/vllm-project/vllm/pull/59135
- @www6v made their first contribution in https://github.com/vllm-project/vllm/pull/59749
- @xiaofei-zheng made their first contribution in https://github.com/vllm-project/vllm/pull/55104
- @xiaoyu-xyz made their first contribution in https://github.com/vllm-project/vllm/pull/55161
- @xuantengh made their first contribution in https://github.com/vllm-project/vllm/pull/56740
- @yantinglai made their first contribution in https://github.com/vllm-project/vllm/pull/59530
- @YashasviAsthana made their first contribution in https://github.com/vllm-project/vllm/pull/59020
- @YidaWeng made their first contribution in https://github.com/vllm-project/vllm/pull/59373
- @yongqiw-i made their first contribution in https://github.com/vllm-project/vllm/pull/59159
- @YueshenZ made their first contribution in https://github.com/vllm-project/vllm/pull/59257
- @YuvrajSinghBhadoria2 made their first contribution in https://github.com/vllm-project/vllm/pull/57035
- @zhangchengzhucufe-dev made their first contribution in https://github.com/vllm-project/vllm/pull/59886
- @ZhenchengLin made their first contribution in https://github.com/vllm-project/vllm/pull/59251
- @ztysth made their first contribution in https://github.com/vllm-project/vllm/pull/59721
Contributors
@BugenZhao, @aoshen02, @stefankoncarevic, @mgoin, @khluu, @hmellor, @yewentao256, @lesj0610, @MatthewBonanni, @jperezdealgaba, @ZJY0516, @AndreasKaratzas, @aarushjain29, @Sunt-ing, @NickLucche, @vadiklyutiy, @hickeyma, @divakar-amd, @yma11, @jzakrzew, @djramic, @simondanielsson, @microslaw, @Thangnguyenvn98, @mjkvaak-amd, @njhill, @chaojun-zhang, @sfeng33, @zhenwei-intel, @andyxning, @matteso1, @vMaroon, @alec-flowers, @simon-veitner-redhat, @TheEpicDolphin, @cagrikymk, @atalman, @shijie-lyu, @cheese-cakee, @Juntian777, @ivanium, @chaunceyjiang, @louie-tsai, @Dovis01, @apex-mochen, @wzhao18, @venkywonka, @pmanczak, @stecasta, @jinzhen-lin, @gty111, @noooop, @Rohan138, @rasmith, @laithsakka, @Fangzhou-Ai, @tlrmchlsmth, @Isotr0py, @cjackal, @vllmellm, @liusy58, @reidliu41, @peperunas, @shimib, @ojeda-e, @yonigozlan, @ShuoleiWang, @lucamotz, @ai-jz, @he-yufeng, @DimensionSTP, @taneem-ibrahim, @LucasWilkinson, @wangshangsam, @whx-sjtu, @micah-wil, @bzsuni, @robertgshaw2-redhat, @arpera, @sricursion, @mawong-amd, @mmastrac, @rmhaskarnvidia, @nvbfalk, @debanshd, @abmfy, @shaohuaxi, @jin-amd, @xijiaat, @ashraf-bhuiyan, @beenpow, @LioEinaudi, @Ronald1995, @R3hankhan123, @ZhengGong-amd, @Akashcodes732, @S1ro1, @yuvalluria, @mganczarenko, @DarkLight1337, @pisceskkk, @YidaWeng, @TQCB, @gau-nernst, @askliar, @benchislett, @jackLei0901, @vx120, @amd-mghanimi, @YueshenZ, @aniskumar-nv, @fululi12, @liuzijing2014, @zyongye, @Prudhvivuda, @kliuae, @wjabbour, @AbhayShuklaIIT, @WarrierRajeev, @Jeff-Tseng-dev, @yifanFengg, @elvircrn, @zupengwang, @tangzzycc, @jeejeelee, @roikoren755, @JohnQinAMD, @LeizhenZhang, @Chinmay-Kulkarni-AMD, @sungbin1015, @zhangchengzhucufe-dev, @fxmarty-amd, @lucianommartins, @nishantrevur, @hyukjlee, @andylolu2, @wangxiyuan, @stu-cao, @vineethsaivs, @zzaebok, @WoosukKwon, @franciscojavierarceo, @hpdkhoa, @gcanlin, @zigzagcai, @ultramancode, @warrendeng, @surajsharan, @subhashpolisetti, @ilmarkov, @awesome-pro, @faaany, @spandantiwari, @almogtavor, @ahmed-bsod, @danigarciaoca, @Alnusjaponica, @NNNtrance, @vaclavcadek, @YashasviAsthana, @QwertyJack, @Ubospica, @AlanFokCo, @jungjiyu, @lcskrishna, @YukioZzz, @hubunt, @sashko-zakharchuk, @staryxchen, @yzong-rh, @Ostring24, @taking-lying-flat, @drakosha, @CorgiBoyG, @QHarshil, @zhou9402, @shiyuan680, @fjosw, @a-sidorova, @GOavi101, @mrodden, @fuscof-ibm, @Taimys, @Rukhaiya2004, @KernelClint, @almayne, @JulienDarve, @sshlyapn, @eldarkurtic, @rbrugaro-amd, @GuyStone, @ZhenchengLin, @ArcheyChen, @djmmoss, @hjjq, @namgyu-youn, @jiangyunfan1, @ganeshr10, @markmc, @yantinglai, @Id545, @roipony, @JadenMathias, @hclsys, @Anurag-M1, @ukannika, @blueberry808, @nv-hwoo, @guanxingithub, @stmatengss, @mohit-sarvam, @Legend-2727, @xiaoyu-xyz, @hdq66666, @Lcy0801, @gaby, @erictang000, @jdebache, @Ayu190505, @DOCCA0, @C1ves, @maeehart, @wangyicong52, @AIwork4me, @Woolgathererer, @yongqiw-i, @divyvasal, @1fanwang, @TyroneNel, @ShengleiFu, @bluemelov1, @kamb-code, @LiRunGuo, @Thiago4532, @harshaljanjani, @ieaves, @NolenLiang, @sudhanshu112233shukla, @abhayjoshi201, @charlesli640, @Sohaib-Ahmed21, @akii96, @notimesea, @JRJRJPRO, @usberkeley, @Mazukiri, @brandonfzhang, @amd-xavierwang, @sawsa307, @czhu-cohere, @wtdcode, @tarukumar, @LostFox11, @big-yellow-duck, @SherifWaly, @roG0d, @esmeetu, @mgehre-amd, @samnordmann, @SunnyR, @ovidiusm, @cleonard530, @eky-amd, @yinli-systems, @MadhavJivrajani, @cquil11, @ywang96, @PatrykSaffer, @PatchouliTIS, @aditi-amd, @git-jxj, @vovani, @MohammadHijjawi97, @TommyTranX, @viktorkou, @xaguilar-amd, @MohitAMD, @danichan-mkm, @pondzikk, @ebfio, @mkhazraee, @ai-yang, @xuantengh, @dhairyashilRG, @www6v, @v-shobhit, @ejrtks1020, @Kylinny, @bnkaiteam, @huishi-hs, @jiahanc, @shen-shanshan, @zxd1997066, @Aravind-11, @usmanmateen, @devshah-cohere, @praveingk, @JartX, @TylerMa-debugg, @lobo235, @dougbtv, @Tialo, @kyleliang-nv, @YeonwooSung, @Kevinli24, @aijanai, @spa5k, @DGlan, @Btwestyo, @xianbaoqian, @rdlh, @hao-aaron, @errmakov, @amd-sriram, @ntheanh201, @LCAIZJ, @damienlaine, @shreyanshj10, @ischencheng, @0xViking, @yisustc, @bigPYJ1151, @Roy214, @Gregory-Pereira, @YuvrajSinghBhadoria2, @surawut-jirasaktavee, @Oxygen56, @andakai, @BolinSNLHM, @Dao007forever, @siliangchen-amd, @xiaofei-zheng, @gc-fu, @puneetsharma21, @linitra24, @devpramod, @yankay, @hlin99, @Godmook, @bvolpato, @UNIDY2002, @tc-mb, @JaredforReal, @hubert-marek, @Zhenzhong1, @AMD-yanfeiwang, @thuongvu, @JimmyWang0417, @MatthewCYM, @i-m-aditya, @lianakoleva, @peakcrosser7, @hoaivoke, @njriasan, @tjtanaa, @haotiaa, @superdiaodiao, @josh-schell-amd, @ztysth, @sungsooha, @Etelis, @natsala13, @ZhouXiaoya25
v0.31.0
Highlights
This release features 717 commits from 307 contributors (96 new)!
- DeepSeek-V4.1-Flash performance: FlashMLA mega attention with the V4.1 NVFP4 compressed KV cache is now the SM100 default (#56935); DeepGEMM sparse MQA logits for the indexer (#56254) and Mega-Gate fusing the gate GEMM with expert selection (#56266); decoder boundaries fuse the TP all-reduce, mHC input preparation (#57643) and the MoE finalize (#58586); a fused small-batch WO-A with inverse RoPE and MXFP8 quant on SM100/SM103 (#58634); the MXFP8
wo_bGEMM fused with the sequence-parallel reduce-scatter (#57428); Engramwkvsharded across TP ranks (#58678) and Engram host tables shared across co-located DP replicas by default (#57651); encoder CUDA graphs for the vision tower (#56625); and SWA bounded replay that keeps the sliding-window KV out of prefix caching (#56227). - Fast restart: the new
vllm preloadCLI launches the weight-cache daemon that keeps post-quantized weights resident in GPU memory across engine restarts (#56680), now with data parallelism (#57386), MTP draft models (#57312), a/healthendpoint (#58552) and a readiness wait (#58370). Experimental initialized-engine snapshots (vllm snapshot create/restore) use CRIU to restore a fully initialized TP1 engine (#51360). - Model Runner V2 and speculative decoding: draft-model speculative decoding (#43091) and custom logits processors (#56497) on Model Runner V2; the new LiLiCorr drafter (#57934); async scheduling for DFlash (#58065) with the context K/V precompute captured in the draft CUDA graph (#57632); DSpark adaptive verification for Gemma4 (#57263) and variable-length decode for Kimi-K3 (#52988); a DCP target with a non-DCP DSpark draft (#56723); and MoE memory now counted during MRV2 profiling, avoiding OOMs on WideEP deployments (#57270, #58411).
- Large scale serving: MoonEP balanced EP all2all backend via
--all2all-backend moonep(#52101), prefill context parallelism with data parallelism (#57075), a low-SM multimem reduce-scatter for SM100/SM103 (#55072), DeepEPv2 with sequence parallelism (#57210) and EPLB with shared-expert overlap (#57236), a sharding-aware NCCL M2N weight-transfer backend for RL (#51520), and KV offloading back-pressure detection (#50045). - Scheduling controls:
--max-num-active-seqscaps RUNNING admission independently ofmax_num_seqs(#56758),--long-prefill-token-thresholdnow adapts to the number of waiting prefills instead of chunking a lone request (#57951, #58459), the waiting queue was reworked so requests already holding KV blocks are scheduled first (#58947), and the KV connector + MTP deadlock under KV pressure was fixed (#57104). - HiSparse hardening: MTP verification rows resolved with a union residency kernel (#59235), MTP acceptance collapse under FULL graphs fixed (#59309), no GPU pages without host backing (#59036), a chunked-prefill preemption livelock fixed (#59494), KV cache sized from the groups HiSparse allocates (#59450), and host prefix publication and GPU prefix adoption fixes (#59007, #59282).
- Security: per-request
mm_processor_kwargsandmedia_io_kwargsare rejected unless--trust-request-mm-kwargsis set (#58830); prefix-cache extra keys are tagged by source so a LoRA name and acache_saltcan no longer collide (#51899), and the LoRA path is part of the block hash (#59335); stale multimodal receiver-cache entries can no longer replace fresh payloads (#57833). - Breaking changes: per-request multimodal kwargs gated (#58830);
tokenizer_mode="slow"removed (#58545);--enable-mamba-fine-grained-prefix-cacherenamed to--enable-mamba-shared-prefix-checkpoint(#57382); online quantization throughquantization="fp8"replaced by thefp8_per_tensorshorthand (#53585) and Quark silent online quantization removed (#51800); the AllSpark INT8 W8A16 backend removed (#58001);--enforce-eagernow also disables JIT kernel warmup (#58197); XPU graphs enabled by default withVLLM_XPU_ENABLE_XPU_GRAPHremoved (#51600).
Release Artifacts
Python Wheels
| Platform | Install |
|---|---|
| PyPI (CUDA 13.0) | pip install vllm |
| PyPI (CUDA 13.0, uv) | uv pip install vllm --torch-backend=auto |
| ROCm | pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.31.0/rocm723 |
| XPU | uv pip install vllm --extra-index-url https://wheels.vllm.ai/0.31.0/xpu --extra-index-url https://download.pytorch.org/whl/xpu --index-strategy unsafe-best-match |
Docker Images
| Platform | Docker Image |
|---|---|
| CUDA 13.0 (Default) | docker pull vllm/vllm-openai:v0.31.0 |
| CUDA 12.9 | docker pull vllm/vllm-openai:v0.31.0-cu129 |
| ROCm | docker pull vllm/vllm-openai-rocm:v0.31.0 |
| CPU | docker pull vllm/vllm-openai-cpu:v0.31.0 |
| XPU | docker pull vllm/vllm-openai-xpu:v0.31.0 |
Other Artifacts
Pre-built release artifacts are available in the Assets section at the bottom of this page, including:
- Source distribution tarball
- CUDA 12.9 Python wheels for x86_64 and arm64
- CUDA 13.0 Python wheels for x86_64 and arm64
- CPU Python wheels for x86_64, arm64, and macOS
- XPU Python wheel for x86_64
Model Support
- New model capabilities: DiffusionGemma structured generation mode with bounded single-token choices (#57250), MiMo V2 MXFP4 MoE, BF16 MoE router and DFlash drafts (#57784), Cohere2MoE auxiliary hidden states for EAGLE3/DFlash drafters (#49819), GLM-5.2-MXFP4 on the ROCm DeepSeek-V3.2 path (#51915), AMD-Quark mixed-precision DeepSeek-V4.1-Flash-MXFP4 (#57071) and GLM-5.3-Flash Quark MXFP4 (#56176) checkpoints, and a built-in
granite_thinking_parserfor Granite 4.2 (#55957). - DeepSeek-V4.1-Flash: FlashMLA mega attention with the NVFP4 compressed KV cache as the SM100 default (#56935), DeepGEMM sparse MQA logits in the indexer (#56254), Mega-Gate (#56266), fused TP all-reduce + mHC input preparation (#57643) with the MoE finalize folded in (#58586), fused small-batch WO-A (#58634), MXFP8
wo_bGEMM + reduce-scatter (#57428), overlapped mHC coefficients for small TP batches (#57603) limited to FULL CUDA graphs (#57874), Engramwkvsharded across TP (#58678), serialized Engram lookups with huge-page host tables (#56926), Engram tables shared across DP replicas by default (#57651, #57914, #59068), native shared-expert MegaMoE fusion without padding (#56568, #57204), faster MegaMoE staging and NVFP4 cache gathers (#57604), the fused query RMSNorm + MXFP8 path restored (#57679), KV-only DSpark context insertion (#56441), vision encoder CUDA graphs (#56625, #58499), SWA bounded replay (#56227), causal image SWA restored to match the reference (#57152), updated reasoning effort mappings (#58316), and startup fixes for NaN-scored candidate blocks (#57454), DeepSelect sentinels (#58215), DSpark non-causal attention on FlashInfer (#57432), runtime JIT of offset candidate buffers (#57667) and imports without Triton (#57654). - DeepSeek V4: FlashInfer sparse MLA with fused inverse RoPE + FP8 quant (#58621), stacked DSpark context WKV projections (#54674), fused MoE expert distribution computed from the EP group (#57465), FIM completion via
suffix(#44229), missingstring=in tool calls parsed (#56271), request tools attached to an existing system message (#51856), image block spacing preserved (#56882), and DSpark width separated from MTP stage validation (#54631). - GLM-5.3-Flash: opt-in FlashAttention and FlashMLA sparse backends on SM90 (#55385), NoPE sparse MLA on the FlashInfer SM120 backend (#55277), cooperative top-k for small decode batches (#57327), indexer decode workspace sized by pooled length (3 GiB saved, #57701), metadata ops 1.6-4.8x faster (#58450), fused kpool tail slot mapping (#57534), kpool top-k through the shared dispatcher (#57546), lower sparse MLA preparation overhead (#57458), no D2H sync in the SM90 sparse MLA plan under async scheduling (#58684), FlashKDA keeping the recurrent state in FP32 for long prefills (#58846), and correctness fixes for kpool corruption with speculative decoding (#58454), SM90
index_kpoolmismatch (#58704), kpool tail strides (#57477), 500k-token prompts (#57317), dense MLP layers under sequence parallelism (#58061), indexer top-k backend selection (#58594) and varlen paged MQA launches (#55270). - Qwen3.8-Flash-Next (Qwen4Exp): FP8 main KV cache on the QSA path (#55557), FP8 TP with FlashInfer TRTLLM MoE (#55867), SM90 QSA tuning (#57273), fused HC down projection + SiLU (#58957), lower PLE metadata overhead (#58114), QSA indexer workspace fragmentation fixed (#57105), profiling KV cache released (#58961), pinned PLE prefetch ids kept out of the CUDA graph pool (#58489), and indexed expert-mapping lookups saving about 25 s of weight loading on DGX Spark (#58720).
- Kimi K3 and MiniMax-M3: Kimi-K3 vision patch embedder as a GEMM (#58527), fused KimiViT QK RoPE (up to 29x, #58651), routed expert quantization (#57430), reasoning parser (#57098) and reasoning token counting (#58372) fixes, DSpark context KV pointers refreshed after re-binding (#58814), stateless first chunks no longer classified as decodes (#51483), and Mamba block estimates that no longer stall admission on external prefix hits (#57050); MiniMax-M3 encoder CUDA graphs (#58673),
Conv3dLayerpatch embedding (about 62x, #58512),triton_mropein the vision tower (#58526), fused MiniMax2 routing with non-unit scaling (#58880), and processor fixes (#58460, #59613). - DiffusionGemma: one-pass sampler statistics kernel (#58226),
diffusion_constrainedreads overlogprob_token_ids(about 25% faster, #58216), fewer logit rows for prefill-only batches (#57416),logprob_token_idssupport (#57417), and fixes for concurrent logprobs (#57414), eager fallback dtype (#57462), multimodal inputs (#57589), quantized LM heads (#48521) and CPU execution (#58964). - Multimodal: Triton
mm_input_normkernel (#56798, #56711), raw pixels kept through the DP-sharded ViT path (#56872), Qwen2.5-VL video fps honored for temporal M-RoPE (#47736), Whisper and Qwen2-Audio clips longer than 30s (#57769, #56912), Mistral3 placeholder grid (#53758), Idefics3 unsplit patches (#48760), Ovis2.5 tokens (#52623), Aria expert weights (#57487), MiMo-V2.5 fused FP8qkv_projsharding (#57508), Gemma4AutoWeightsLoader(#55911) and buffer scalars (#54213), Gemma4 FP8 KV with FA4 at head dim 512 (#53175), Laguna RoPE (#57189), Mistral-Large-3 accuracy regression (#57563), malformed EXIF (#56527, #57234), JinaVL labels (#57347), Sarvam MLA routing with FP32 router logits (#56034), fused CohereASR attention scores (#55190), and fused DFlash2 grouped convolution (#55960). - LoRA: Nemotron VL language models (#56231), ModernBert (#57148), VoyageQwen3 embeddings (#57708), RoBERTa sequence classification (#58884), and
modules_to_savesequence-classification heads (#53555) with per-adapternum_labels(#57766).
Engine Core
- Model Runner V2: draft-model speculative decoding (#43091), custom logits processors (#56497) validated at admission (#57728), randomized dummy inputs (#58411), dummy tokens routed to MoE experts during profiling (#57270), FULL decode graphs for one-token prompt tails (#58400), token-to-request mappings shared (#57102) and Mamba/GDN metadata reused across KV cache groups (#58762), encoder-only ViT CUDA graphs (#56922), weight offloader (#57834) and pooling model (#57737) released on shutdown, and fixes for multi-layer MTP KV in P/D (#55055), fast-prefill with LoRA (#56456), stale block-table writes from dummy draft steps (#56734), padded prompt tails in hybrid models (#58434), never-proposed draft slots (#58784), auto-fit
max_model_len(#58149) andintermediate_tensorsduring capture (#57745). - Speculative decoding: LiLiCorr drafter (#57934), DFlash async scheduling (#58065) and context K/V in the draft CUDA graph (#57632), Gemma4 DSpark adaptive verification (#57263), Kimi-K3 variable-length adaptive verification (#52988), CPU-GPU sync removed for heterogeneous vocabularies (#57396), Triton recompiles avoided in the acceptance estimator (#57107), and fixes for EAGLE/dense drafts with EP (#56930), DFlash/DSpark profiling batches (#56448), GLM MTP head memory (#55442), prompt embeddings with drafts (#57356), and TritonMLA causal multi-token decode (#51065).
- Scheduler:
--max-num-active-seqs(#56758), adaptive--long-prefill-token-threshold(#57951, #58459),skipped_waitingreplaced by a KV-holding waiting queue (#58947), atomic admission ofn > 1requests (#53936), KV connector + MTP deadlock (#57104), a throughput cliff whenmax_num_seqsis not a multiple of 8 (#57355), streaming continuations keeping logprobs and refreshing max tokens (#57447, #57676), a resumable request + async scheduling race (#58259), and a non-blocking structured-output grammar poll (#55931). - Prefix caching and hybrid models: extra keys tagged by source (#51899) and LoRA paths hashed (#59335),
--enable-mamba-shared-prefix-checkpoint(#57382), prompt-tail hits with MTP restored (#58368), prompt-end checkpoints kept under sparse retention (#59146), align-mode checkpoint reservation (#59175), stateless GDN first chunks (#51565), MTP draft KV cache groups annotated on the hybrid grouping path (#55390), a generalized prefill checkpoint builder (#57783), batched Mamba2 prefill state saves without GPU-CPU syncs (#49371),skip_reading_prefix_cachehonored for connector hits (#57269), incremental multimodal block hashing (#51694), a KV block size every attention backend supports (#49845) with clearer errors (#58557), andCacheConfig.effective_attention_block_sizefor DCP (#56538). - Startup and memory: parallel Triton warmup compilation (#58582), JIT warmup disabled under
--enforce-eager(#58197) unless fault tolerance is on (#58593), DeepGEMM warmup reusing the MoE workspace (#57268), allocator fragmentation no longer shrinking the KV cache during profiling (#58430), sparse prefill buffers reserved before KV sizing (#57575), stale FlashMLA workspace views released (#56902), DeepGEMM FP8 workspace halved (#53914), shared Marlin/Humming workspaces (#57421), FlashInfer BF16 MoE weights converted in place (#54699), UniProc startup threads bounded by the CPU quota (#58946), and runtime threads set before profiling (#55891). - Kernels: sampled filtering for persistent top-k (#56346), Murmur3 RNG for Gumbel sampling (#51367), register-resident per-token-group 8-bit quant (#55330), vectorized per-tensor FP8 abs-max (#58194), a Triton kernel dispatcher for platform-specific overrides (#43048),
GateLinearfor all MoE models (#58234), non-local expert slots skipped inTritonExperts(#58051), deferred TRT-LLM-Gen top-k finalize on the modular path (#58635), SM120 batch-invariant matmul configs (#57456), FlashInfer prefill dequant scratch bounded (#57918), and fixes for Triton softcap NaNs (#56579), QuIP Hadamard transforms (#43462), CUTLASS FP8 linear on A100 (#55884), FlashInfer sampling on unsupported GPUs (#48956), SM100 FP8 blockwise scale padding (#57377), mixed FULL graph prefill capture (#58275) Inductor custom-op pattern matching (#58189), CuteDSL BF16 GDN prefill diverging from FLA on Qwen3.5 (#53864), SM100fp8_ds_mlacache scales (#49435), ragged decode batches in the sparse indexer (#52500), staleallowed_token_idsmasks after batch reordering (#48419, #43931), andprompt_embedstensors held after requests finish (#57988). - Batch invariance: breakable CUDA graphs without torch.compile by default under
VLLM_BATCH_INVARIANT(#57586), sequence parallelism and async TP disabled (#56377), and NCCL 2.31 collectives kept enabled (#58179). - Sleep and RL:
release_kv_cache_memory()frees only KV cache memory (#44890),--sleep-preserve-parameter-namesretains frozen weights across level-2 sleep (#57891), NCCL M2N weight transfer (#51520), dense DP weight updates by DP index (#56950), allocator config kept when toggling expandable segments (#57982), and cache reset failures propagated during sleep (#54581). - Fast restart:
vllm preload(#56680), DP (#57386), MTP drafts (#57312), readiness wait (#58370),/health(#58552), ModelOpt MXFP8 pre-processed weights (#57316), and initialized-engine snapshots (#51360). - Logging and observability:
LoggingConfigvia--logging-configand--log-level(#57205), JSON logging fixes (#57957, #58747), startup log suppression fixed (#51366), unified platform-aware torch profiling (#57460), no 0.0% prefix cache hit rate before any query (#54990), KV transfer metrics formatting (#57068), MFU activation sizing from the model dtype (#57070), platform-overridable env var checks (#48599),VLLM_TARGET_DEVICE=empty pip install vllmfor out-of-tree backends (#41074), and the correct vLLM version reported when installing from source (#57295, #57744).
Large Scale Serving
- MoE communication: MoonEP backend (#52101), DeepEPv2 with sequence parallelism (#57210) and EPLB + shared-expert overlap (#57236), low-SM multimem reduce-scatter (#55072), EPLB load statistics during Elastic EP scaling (#58473), SP padded rows skipped in grouped routing so rank 0 is no longer a prefill straggler (#56079), hash routing rejected on unsupported monolithic backends (#57867), sampled-token broadcasts skipped under PP for requests leaving the engine (#58542), DBO with DeepEP low-latency profiling (#57502), external LB with replicas sharing nodes (#53743), and no blocking RPC during the engine handshake (#57226) or unbounded draft-token waits (#58779).
- Context parallelism: PCP with DP, EP and MTP (#57075), a DCP target with non-DCP DSpark/DFlash drafts (#56723), DCP sequence lengths without a CPU-GPU sync (#58169), and NIXL DCP pulls across MLA cache regions (#57389).
- KV connectors: NIXL pipeline-parallel push prefill for attention-HMA (#50494) and packed MLA layouts (#50499), transport-failure metrics split from KV expiry (#55854), dead peer state released without waiting for TTL (#50047), expired leases reaped behind a heartbeated head (#58292), push completion restored (#58188), D-side activity recorded (#52245); Mooncake
CUSTOM_MEM_POOL(#49300), request-level load failures under HMA (#56855, #57174) and bootstrap retries (#58919); MoRIIO hybrid Mamba/KDA state in READ mode (#51052) and a multi-decode routing race (#51681); prefill cache hits inprompt_tokens_details(#54222); KV-event publishers bound at port 0 (#55844); KV cache metadata GET restored for external consumers (#56925); partial-block KV events keep every multimodal feature (#58288); saves finalized on steps without a forward (#57775); abort-safeExampleHiddenStatesConnector(#56841); DecodeBench FP8 fills (#58472); aKvHintsrequest envelope (#53423); and the AuxOutput connector for routed-expert outputs (#45635, #58150, #58205). - KV offloading: per-request
max_load_tokens(#55885), back-pressure detection (#50045),SimpleCPUOffloadConnectorPrometheus metrics (#57251), non-prefix-cacheable (#56810) and scratch (#57145) groups skipped, replicated layouts for multi-group MLA (#57652), regions of 512 GiB or more registered in chunks (#51081), cgroup memory checked before SHM allocation (#54014), and fixes for cache recency (#51787), MTP-retained sliding windows (#56709), canonical MLA rows (#56799), event metadata (#57453) and ROCm pinned memory (#57160). - HiSparse: union residency kernel for MTP rows (#59235), MTP acceptance under FULL graphs (#59309), host-backed allocation (#59036), preemption livelock (#59494), KV cache sizing (#59450), host prefix publication (#59007), GPU prefix adoption (#59282), and residency metrics (#58725).
- Encoder disaggregation (EPD): dynamic EPD proxy with launcher-managed registration (#54176), image requests batched per encoder (#57095), cross-encoder caching through Mooncake (#56242), metadata-only audio inputs (#57887), language-model shards skipped for
--mm-encoder-only(#58086), EC connector metrics (#54960), and encoder-only fixes (#58490, #58287, #57696).
Hardware & Performance
- NVIDIA: FlashInfer 0.7.0.post1 (#58069, #59323), DeepGEMM pin bump with SM120 fixes (#57218), Rubin CUDA 13.4 nightly images (#55953), and QuTLASS builds with PyTorch 2.13 (#58173).
- AMD ROCm: AITER v0.1.23 (#56885, #58867), torch 2.13 and Triton 3.8 (#50605, #58006),
triton_kernels3.8 MXFP4 MoE for gpt-oss and DeepSeek-V4 (#55934), a ROCR host segfault fixed (#57328); DeepSeek-V4/V4.1 HCA dual-stream (#56853) and layer-aware CSA2 overlap (#57407), FP8wo_a(#54894), inverse RoPE fused into the sparse decode reduce (#57435, #57451) which now emits MXFP8 for a grouped FP8wo_a(#58456), MXFP8 GEMM on native 32x32 scales on gfx950 (#58510), reused top-k ragged metadata (#57434), faster candidate block selection (#58208), Engram tables in host memory (#57491) and Qwen3.8-Flash-Next PLE tables offloaded to host memory (#57497), an opt-in AITER ASM decode route for DCP + speculative decoding (#56861),get_top_tokens()on the DeepSeek V4 MTP drafter (#57568), DSpark adaptive verification (#52362), opt-inVLLM_DSV4_LOGITS_FIXfor sparse-indexer logits on gfx950/gfx942 (#50455), an accuracy revert of #56433 and #51692 (#57132), and clear errors for FSE with DPA+ETP (#57919); Kimi-K3 a4w4 FlyDSL kernels (#53940) withVLLM_ROCM_USE_AITER_MOE_SITUV2=a16w4|a8w4|a4w4(#58201), low-concurrency speculative KDA (#58045), sharded latent MoE under EP (#54956) and fewer projection copies (#50592); a ROCm Hy4 path with torch.compile (13x lower decode latency, #57526); MiniMax-M3 AITER QK-norm fusion (#54535), copy-free K/V insert (#56849) and MXFP8 fixes (#53674, #58089); GLM-5.3-Flash boot fixes (#57192, #57252, #57425); AITER QuickReduce + RMSNorm (#48249), BF16 AsyncTP (#58098), static FP8 attention output fusion (#58099), QK-norm/RoPE/KV-cache fusion for MRoPE (#50212), AITER GDN decode for flat layouts (#53623),wvSplitKfor single-output GEMMs (#53283), 69 fewer copies per decode step on the skinny GEMM path (#58566), tuned GEMM lookup through AITER (#55001), a narrower Triton prefill KV tile on RDNA3/RDNA4 (#58225), staged large pageable H2D copies (#56343), and fixes for MXFP4 MoE padding starving the KV cache (#56359), AITER MLA FP8 prefill OOM (#57923), unquantized cache descales (#56726) and AITER MoE fallbacks (#56590, #57866, #57426).VLLM_ROCM_USE_AITER_FP4_ASM_GEMMis restored and off by default (#57055); SWA bounded replay is disabled on ROCm (#57906). - Intel XPU: PyTorch 2.14 (#56013), XPU graphs on by default (#51600), EPLB (#44987), int8 W8A8 MoE on Triton (#53162), batch invariance (#55881), SYCL rotary embedding (#55721), fused QK RMSNorm + RoPE + gate (decode region 55% faster, #56096), Model Runner V2 sampler (#57277) and PP microbatch control (#55145),
--device-idshonored (#56015), and device pointer overflow fixed (#54514). - CPU: FP8 W8A8 linear and MoE for Intel Diamond Rapids (#49942), Arm paged attention up to 25% faster (#56045), Zen DA8W4 int4 for dense and MoE layers (#54024), zentorch SDPA for encoder attention (#54508) and MLA prefill (#54967), FP32 attention sinks (#56252), W8A8 INT8 MoE on POWER (#55316), W4A16 Whisper (#58268), wheels built on Ubuntu 22.04 (glibc 2.34) with AMX-FP8 (#58515), AVX10.2 gated on compiler support (#58133), pre-built Triton CPU (#58140),
--device-memory-utilizationalias (#56547), vLLM Recipes in the CPU image (#58796, #57306), s390x protobuf pin (#54978) and torchcodec video (#58693), and fixes for Ministral FP8 (#56985), FP32 router weights (#56168), zentorch import failures (#54923), macOS multimodal SHM (#57142), CPU affinity per local rank (#53636) and NIXL GDN state layout (#53300).
Quantization
- Humming: Hadamard transforms and NVFP4/MXFP4/MXFP8 online quantization (#56685), asymmetric wNaM through compressed-tensors (#46528), humming-kernels 0.1.16 (#58054), and Humming in the W4A8 (INT4xFP8) MoE oracle (#58427).
- New capabilities: native Quark W4A16 INT4/UINT4 exports (#48606), opt-in load-time MXFP4 dequantization (#50814), explicit per-token NVFP4 MoE backends (#57176), and a canonical N-first layout for compressed-tensors WNA16 MoE (#52798).
- Fixes: MXFP8 on layers below
mm_mxfp8shape limits (#54223), online NVFP4 scales on reload (#57954), LM head linear metadata (#58444), and the fused SiLU-mul block-quant path skipped under a SwiGLU clamp (#57984).
API & Frontend
- New options and endpoints:
--tool-strict-level(#56268),response_formatwithtool_choice=auto(#56086), DeepSeek-V4 FIM completions (#44229),release_kv_cache_memory()andPOST /release_kv_cache_memory(#44890), fixed-token prefill scoring viaprompt_logprob_token_ids(#54335), per-request speculative decoding metrics in/inference/v1/generate(#43310), per-request metrics (#55084) andcache_write_tokens(#57222) in the Responses API, Anthropicthinkingin/v1/messages(#58613), streaming reasoning and tool calls from the derender endpoint (#50550) with offloaded detokenization (#57528),vllm chatstreaming thinking output (#57045), request body debug logging with--enable-log-requests(#58163), and only the summary line of config docstrings in--help(#57357). - Structured output and parsers: native Lark grammars in the xgrammar backend (#58321), XGrammar 0.2.7 (#57272), Granite migrated to the streaming Parser Engine (#49648), and fixes for reasoning boundaries (#56635), xgrammar choices with control characters (#48115), list-typed JSON Schema (#48416), empty
structural_tag(#47450), outlines EOS handling (#58612, #57743), parser-suppressed streaming logprobs (#58583), Inkling tool names after reasoning (#58792), and lengthfinish_reasonfor truncated streaming tool calls (#46303). - OpenAI, Anthropic and Harmony compatibility: reasoning token counts for Harmony, DeepSeek-V3 and Step3 (#58626) and per Responses tool round (#58927), Harmony
max_output_tokensin the tool loop (#58551), batched chat completions using the adjusted requests (#58929, #58958), a fresh parser per choice (#58939), deferred reasoning recounts (#56067), Responses MCP cleanup (#56988) and imagedetaildefault (#57241), Anthropic inline system detection (#58754) and disabled thinking with P/D (#58786), stop strings rejected on--tokens-onlyservers (#57058), invalidprompt_embedsreturning 400 (#55451, #57006), prompts bounded after multimodal expansion (#57076),--override-generation-configpenalties (#50769), Hub revisions (#56092, #57461), full logprobs in token-in/token-out responses (#58488), generative scoring cancellation (#57729, #58788), gRPC keepalive pings (#55102), andrun-batchdiarized transcriptions (#57948). - Pooling: chunked embedding padding (#56505) and normalization (#57498), reranker tokenization with document limits (#57666), and BERT-family heads kept for raw logits (#57664).
- Rust frontend:
--hf-overrides(#56931),--sse-keep-alive-interval(#58306), custom chat roles (#58311), MiMo V2.5 parsers (#57933), normalized reasoning controls (#56998), parser-owned output grammars (#55269, #57340), sampling masks over gRPC (#56777), local DP size in gRPC metadata withvllm-proto0.3.0 (#57116, #57233), a per-request preemption histogram (#57033), lock-free histograms (#58574), Nemotron-H vision context (#57634), model-owned vision processors (#58109), anmm-processorbenchmark (#51922, #58084, #58378), unsupported serve args recognized (#58330), NaN logprobs no longer killing the engine client (#51026), appendedEngineCoreOutputfields accepted (#56533), andvllm-rsonPATHin the CUDA image (#57606). - Benchmarks: an
openai-responsesbackend forvllm bench serve(#54628) andmodel_idin latency/throughput JSON (#58112).
Security
- Per-request
mm_processor_kwargsandmedia_io_kwargsare rejected by default; trusted deployments opt in with--trust-request-mm-kwargs(#58830). - Prefix-cache block hashes tag extra keys by source (#51899) and include the LoRA path (#59335); multimodal hash input is framed (#54283) and incremental block hashing covers every overlapping feature (#51694).
- Fresh multimodal payloads take precedence over a stale receiver cache (#57833), and encoder-cache hits with mismatched embedding counts are rejected (#57696).
min_tokensabove the filledmax_tokensdefault is rejected instead of wedging the engine (#57731); message sanitization filters upper-case memory addresses (#58832); LoRA adapters named after a served model are rejected (#59286).
Dependencies
- FlashInfer 0.7.0.post1 (#58069, #59323), Transformers 5.17.0 (#56108) with an upper bound in requirements (#59614), XGrammar 0.2.7 (#57272),
oss-harmonyreplacingopenai-harmony(#55128), DeepGEMM fork pin bump (#57218), FlashKDA bump (#58846), and humming-kernels 0.1.16 (#58054). - CUDA 12 images use LMCache 0.4.4 and CuPy CUDA 12 (#57945); a separately tagged zstd Docker Hub image (
-x86_64-zstd) is published (#55608); Rubin CUDA 13.4 nightly images (#55953). - ROCm: AITER v0.1.23 (#58867), torch 2.13, Triton 3.8 (#50605, #58006), patched ROCR (#57328), LMCache OpenTelemetry pins (#59056).
- XPU: PyTorch 2.14 (#56013). CPU: wheels built on Ubuntu 22.04 (#58515), pre-built Triton CPU (#58140).
Breaking Changes & Deprecations
- Per-request
mm_processor_kwargsandmedia_io_kwargsnow return an error unless the server is started with--trust-request-mm-kwargs; server-level--mm-processor-kwargs/--media-io-kwargsand offlineLLMare unchanged (#58830). tokenizer_mode="slow"was removed; it already behaved like"hf"under Transformers v5 (#58545).--enable-mamba-fine-grained-prefix-cachewas renamed to--enable-mamba-shared-prefix-checkpoint(#57382).- Online quantization through
quantization="fp8"now redirects to thefp8_per_tensoronline shorthand (#53585); Quark-specific silent online MXFP4 quantization was removed in favor of the online quantization API (#51800). - The AllSpark INT8 W8A16 GEMM backend was removed (#58001). The
return_assistant_tokens_maskoption of/renderand theassistant_tokens_maskresponse field were removed (#57520). VLLM_PLE_CPU_OFFLOADwas removed; use--engram-config(#57937).VLLM_XPU_ENABLE_XPU_GRAPHwas removed and XPU graphs are on by default (#51600).- The comma-separated form of
--collect-detailed-traceswas removed; use the list syntax (#55702). - New defaults:
--enforce-eageralso disables JIT kernel warmup unless fault tolerance is enabled (#58197, #58593);VLLM_BATCH_INVARIANT=1uses breakable CUDA graphs without torch.compile (#57586) and disables sequence parallelism and async TP (#56377); DeepSeek-V4.1 SWA bounded replay is on (#56227) except on ROCm (#57906); Engram host tables are shared across co-located DP replicas when possible (#57651); FlashMLA mega attention is the DeepSeek-V4.1 default on SM100 (#56935); the AITER w4a4 ASM GEMM is off by default on ROCm (#57055). - Model Runner V1 + PP > 1 + async scheduling + structured output is now rejected at startup (#56250);
json_objectis rejected at validation with the outlines backend (#57743). transformersnow has an upper bound in requirements (#59614).
New Contributors
- @0z5a made their first contribution in https://github.com/vllm-project/vllm/pull/56441
- @200lz made their first contribution in https://github.com/vllm-project/vllm/pull/55528
- @AARONKANG04 made their first contribution in https://github.com/vllm-project/vllm/pull/48521
- @acsoto made their first contribution in https://github.com/vllm-project/vllm/pull/57314
- @adenzhou1350 made their first contribution in https://github.com/vllm-project/vllm/pull/56882
- @adtygan made their first contribution in https://github.com/vllm-project/vllm/pull/56325
- @amasen02 made their first contribution in https://github.com/vllm-project/vllm/pull/56576
- @amd-sriram made their first contribution in https://github.com/vllm-project/vllm/pull/53792
- @andrewor14 made their first contribution in https://github.com/vllm-project/vllm/pull/52956
- @Ankit-Jaiswal-AMD made their first contribution in https://github.com/vllm-project/vllm/pull/56252
- @baljinderhothi-cohere made their first contribution in https://github.com/vllm-project/vllm/pull/58792
- @BaoYunkai made their first contribution in https://github.com/vllm-project/vllm/pull/57568
- @blipbyte made their first contribution in https://github.com/vllm-project/vllm/pull/55612
- @BPbruce made their first contribution in https://github.com/vllm-project/vllm/pull/46466
- @chuan932 made their first contribution in https://github.com/vllm-project/vllm/pull/55330
- @coderfornow made their first contribution in https://github.com/vllm-project/vllm/pull/54978
- @CZT0 made their first contribution in https://github.com/vllm-project/vllm/pull/51694
- @devtyagi3909 made their first contribution in https://github.com/vllm-project/vllm/pull/56635
- @dilberx made their first contribution in https://github.com/vllm-project/vllm/pull/57076
- @divyvasal made their first contribution in https://github.com/vllm-project/vllm/pull/49845
- @equiluxe made their first contribution in https://github.com/vllm-project/vllm/pull/48115
- @errmakov made their first contribution in https://github.com/vllm-project/vllm/pull/58583
- @farzad-elastix made their first contribution in https://github.com/vllm-project/vllm/pull/56168
- @freyfwt made their first contribution in https://github.com/vllm-project/vllm/pull/51367
- @garrett361 made their first contribution in https://github.com/vllm-project/vllm/pull/57984
- @git-jxj made their first contribution in https://github.com/vllm-project/vllm/pull/55702
- @gokay-ai made their first contribution in https://github.com/vllm-project/vllm/pull/58292
- @gongwei-130 made their first contribution in https://github.com/vllm-project/vllm/pull/55102
- @haosenwang1018 made their first contribution in https://github.com/vllm-project/vllm/pull/58288
- @harshit-sarvam made their first contribution in https://github.com/vllm-project/vllm/pull/56034
- @HieDean made their first contribution in https://github.com/vllm-project/vllm/pull/57356
- @huthvincent made their first contribution in https://github.com/vllm-project/vllm/pull/48419
- @i-m-aditya made their first contribution in https://github.com/vllm-project/vllm/pull/57447
- @jayzuccarelli made their first contribution in https://github.com/vllm-project/vllm/pull/58557
- @jiakangkangfuzhe made their first contribution in https://github.com/vllm-project/vllm/pull/55146
- @jiangLLM made their first contribution in https://github.com/vllm-project/vllm/pull/57487
- @jiaran-king made their first contribution in https://github.com/vllm-project/vllm/pull/56242
- @jz-yolo made their first contribution in https://github.com/vllm-project/vllm/pull/58884
- @kaijunli-infr made their first contribution in https://github.com/vllm-project/vllm/pull/52101
- @karya0 made their first contribution in https://github.com/vllm-project/vllm/pull/57453
- @KEYS-A15 made their first contribution in https://github.com/vllm-project/vllm/pull/57528
- @kwen2501 made their first contribution in https://github.com/vllm-project/vllm/pull/51520
- @laulopezreal made their first contribution in https://github.com/vllm-project/vllm/pull/57189
- @lijipeng787 made their first contribution in https://github.com/vllm-project/vllm/pull/58225
- @limitmhw made their first contribution in https://github.com/vllm-project/vllm/pull/48606
- @linnea-lin-00638949 made their first contribution in https://github.com/vllm-project/vllm/pull/47450
- @LinzeShi made their first contribution in https://github.com/vllm-project/vllm/pull/57347
- @mahird3 made their first contribution in https://github.com/vllm-project/vllm/pull/57708
- @melcheikh made their first contribution in https://github.com/vllm-project/vllm/pull/48956
- @MichaelLapshin made their first contribution in https://github.com/vllm-project/vllm/pull/57396
- @mjkvaak-amd made their first contribution in https://github.com/vllm-project/vllm/pull/48249
- @mkunredd made their first contribution in https://github.com/vllm-project/vllm/pull/58244
- @mmastrac made their first contribution in https://github.com/vllm-project/vllm/pull/57414
- @mustafayildirim made their first contribution in https://github.com/vllm-project/vllm/pull/57252
- @Navjot10 made their first contribution in https://github.com/vllm-project/vllm/pull/55390
- @olka-amd made their first contribution in https://github.com/vllm-project/vllm/pull/51065
- @PeganovAnton made their first contribution in https://github.com/vllm-project/vllm/pull/55128
- @Rakul-Chauhan made their first contribution in https://github.com/vllm-project/vllm/pull/54967
- @Ricardo-M-L made their first contribution in https://github.com/vllm-project/vllm/pull/52580
- @Ronnie-Rui made their first contribution in https://github.com/vllm-project/vllm/pull/54581
- @RyanMa29 made their first contribution in https://github.com/vllm-project/vllm/pull/57295
- @sammaji made their first contribution in https://github.com/vllm-project/vllm/pull/58626
- @sawsa307 made their first contribution in https://github.com/vllm-project/vllm/pull/56497
- @ScarWar made their first contribution in https://github.com/vllm-project/vllm/pull/49435
- @sdougbrown made their first contribution in https://github.com/vllm-project/vllm/pull/49819
- @semerandre made their first contribution in https://github.com/vllm-project/vllm/pull/55557
- @sergiofigueras made their first contribution in https://github.com/vllm-project/vllm/pull/57948
- @shallow10 made their first contribution in https://github.com/vllm-project/vllm/pull/58336
- @SIDDARTHAREDDY8 made their first contribution in https://github.com/vllm-project/vllm/pull/57743
- @simpleqt made their first contribution in https://github.com/vllm-project/vllm/pull/55936
- @tangzzycc made their first contribution in https://github.com/vllm-project/vllm/pull/53283
- @touch869 made their first contribution in https://github.com/vllm-project/vllm/pull/55844
- @tripathiarpan20 made their first contribution in https://github.com/vllm-project/vllm/pull/57140
- @twu3202 made their first contribution in https://github.com/vllm-project/vllm/pull/57769
- @ubwzwd made their first contribution in https://github.com/vllm-project/vllm/pull/55931
- @UNIDY2002 made their first contribution in https://github.com/vllm-project/vllm/pull/58370
- @valarLip made their first contribution in https://github.com/vllm-project/vllm/pull/58659
- @vcave made their first contribution in https://github.com/vllm-project/vllm/pull/51681
- @voidxb made their first contribution in https://github.com/vllm-project/vllm/pull/49300
- @vorapolsiloai made their first contribution in https://github.com/vllm-project/vllm/pull/50212
- @vschandramourya made their first contribution in https://github.com/vllm-project/vllm/pull/58779
- @weitliao made their first contribution in https://github.com/vllm-project/vllm/pull/54535
- @wenjinhust made their first contribution in https://github.com/vllm-project/vllm/pull/51366
- @Willian-Zhang made their first contribution in https://github.com/vllm-project/vllm/pull/58720
- @wtdcode made their first contribution in https://github.com/vllm-project/vllm/pull/56268
- @xinnywinne made their first contribution in https://github.com/vllm-project/vllm/pull/55084
- @YannikHinteregger made their first contribution in https://github.com/vllm-project/vllm/pull/50047
- @YashasviChaurasia made their first contribution in https://github.com/vllm-project/vllm/pull/58112
- @Yatimai made their first contribution in https://github.com/vllm-project/vllm/pull/43462
- @YCH188 made their first contribution in https://github.com/vllm-project/vllm/pull/57190
- @yousafshah made their first contribution in https://github.com/vllm-project/vllm/pull/55957
- @ys2025-AI made their first contribution in https://github.com/vllm-project/vllm/pull/56841
- @yuchenwang3 made their first contribution in https://github.com/vllm-project/vllm/pull/54699
- @zhecfy made their first contribution in https://github.com/vllm-project/vllm/pull/58316
- @Zoe923 made their first contribution in https://github.com/vllm-project/vllm/pull/53864
- @Zyann7 made their first contribution in https://github.com/vllm-project/vllm/pull/57784
Contributors
@AndreasKaratzas, @khluu, @mgoin, @njhill, @BugenZhao, @stefankoncarevic, @robertgshaw2-redhat, @taneem-ibrahim, @Thangnguyenvn98, @hmellor, @Juntian777, @WoosukKwon, @yewentao256, @gau-nernst, @mmastrac, @LucasWilkinson, @Fangzhou-Ai, @NickLucche, @aoshen02, @JaredforReal, @mawong-amd, @DarkLight1337, @zyongye, @sfeng33, @vllm-agent, @okorzh-amd, @ZJY0516, @shen-shanshan, @gty111, @Isotr0py, @djramic, @zixi-qi, @wzhao18, @alec-flowers, @aarushjain29, @Rohan138, @yma11, @mjkvaak-amd, @ivanium, @yisustc, @MatthewBonanni, @wangxiyuan, @reidliu41, @chaojun-zhang, @shaohuaxi, @gcanlin, @ganeshr10, @linitra24, @yzong-rh, @divakar-amd, @atalman, @ShuoleiWang, @simondanielsson, @rasmith, @chaunceyjiang, @micah-wil, @liusy58, @LioEinaudi, @louie-tsai, @LiuYinfeng01, @jeejeelee, @zhenwei-intel, @jperezdealgaba, @hlin99, @zxd1997066, @lucamotz, @lucifer1004, @eopXD, @ashraf-bhuiyan, @TheEpicDolphin, @JulienDarve, @mayuyuace, @danisereb, @bigPYJ1151, @Etelis, @JohnQinAMD, @jiangkuaixue123, @tianmu-li, @akii96, @vllmellm, @RyanMa29, @afriedri, @elvircrn, @mustafayildirim, @yuzhouo7, @wtdcode, @biswapanda, @adtygan, @hickeyma, @itayalroy, @jiangLLM, @Hotragn, @faaany, @xhx1022, @jinzhen-lin, @sheralskumar, @Wauplin, @matteso1, @wjabbour, @Sunt-ing, @arpera, @hclsys, @S1ro1, @HDCharles, @fxmarty-amd, @liuzijing2014, @UNIDY2002, @samuelkim7, @markmc, @errmakov, @zhejiangxiaomai, @KernelClint, @i-m-aditya, @ColinZ22, @ppalanga, @franciscojavierarceo, @maithilijoshi20, @mfylcek, @almersawi, @albertoperdomo2, @0z5a, @jacklin78911-collab, @lzhan011, @Alex-ai-future, @freyfwt, @ZhengGong-amd, @mindungil, @amasen02, @devtyagi3909, @wenjinhust, @jbyczkow, @AdaAibaby, @harshit-sarvam, @vineethsaivs, @tripathiarpan20, @sashko-zakharchuk, @semerandre, @rebklee, @git-jxj, @cjackal, @xinnywinne, @andrewor14, @kyleliang-nv, @thillai-c, @dongluw, @ChuanLi1101, @omerpaz95, @jiaran-king, @liuyao0322, @coderfornow, @drakosha, @laulopezreal, @melcheikh, @shantipriya-amd, @Rukhaiya2004, @yuchenwang3, @lxy-alexander, @tlrmchlsmth, @HieDean, @Zoe923, @YCH188, @amd-sriram, @andylolu2, @MicheleCampi, @sdougbrown, @vMaroon, @shimib, @andakai, @thegoldenflow, @Levius-Fubuki, @olka-amd, @limitmhw, @yuwenzho, @adenzhou1350, @Josephasafg, @kylesayrs, @jdebache, @kaijunli-infr, @afierka-intel, @frida-andersson, @bnellnm, @ys2025-AI, @waizuichougou, @touch869, @rjrock, @sawsa307, @pavelzak, @ScarWar, @fadara01, @Mi-Jiazhi, @dilberx, @mahird3, @Zyann7, @roikoren755, @YukioZzz, @Ronnie-Rui, @acsoto, @czhu-cohere, @twu3202, @oliverholworthy, @weitliao, @sergiofigueras, @tuukkjs, @Sip4818, @karen-sy, @garrett361, @GirasoleY, @Navjot10, @taking-lying-flat, @wangyicong52, @majunze2001, @gangula-karthik, @ubwzwd, @linnea-lin-00638949, @kushaldabbe, @Yatimai, @guanxingithub, @jiakangkangfuzhe, @MichaelLapshin, @jhu960213, @tpopp, @mganczarenko, @qiching, @tangzzycc, @lijipeng787, @nikhilkulkarni1755, @QwertyJack, @Ankit-Jaiswal-AMD, @vorapolsiloai, @Dao007forever, @BPbruce, @divyvasal, @simpleqt, @blipbyte, @jhaotingc, @karya0, @farzad-elastix, @grYe99, @talorabr, @jackLei0901, @Yejing-Lai, @Priyjain-amd, @KEYS-A15, @LinzeShi, @bohnstingl, @nightcityblade, @vcave, @AARONKANG04, @CZT0, @snadampal, @microslaw, @haosenwang1018, @netanel-haber, @khushali9, @sammaji, @SIDDARTHAREDDY8, @100milliongold, @gongwei-130, @fululi12, @mkunredd, @mrodden, @noooop, @valarLip, @shallow10, @200lz, @askliar, @chuan932, @Monishver11, @YashasviChaurasia, @gokay-ai, @jiacao-amd, @yousafshah, @positive666, @TQCB, @lk-chen, @Ricardo-M-L, @huthvincent, @baljinderhothi-cohere, @vschandramourya, @kwen2501, @xiao-llm, @zhecfy, @Willian-Zhang, @simon-veitner-redhat, @mevince, @voidxb, @hongxiayang, @YannikHinteregger, @V-3604, @Rakul-Chauhan, @frankwang28, @LCAIZJ, @PeganovAnton, @jz-yolo, @QHarshil, @R3hankhan123, @jayzuccarelli, @yannicks1, @BaoYunkai, @xaguilar-amd, @xiaohuguo2023, @wxsIcey, @harshaladhav-amd, @varun-sundar-rabindranath, @kliuae, @LostFox11
v0.30.0
Highlights
This release features 762 commits from 315 contributors (104 new)!
- New models: DeepSeek-V4.1-Flash (#56214, #56228, #56208) with the whole KV stored in MXFP8 through the FlashMLA V4.1 record on SM100 (#56893), DeepGEMM Mega-mHC (#56962), and async Engram prefetch with Engram DP sharding (#56512); DeepSeek-V4-Flash-Vision-Exp (#54566), also on ROCm (#55107) and with LoRA (#55897); GLM-5.3-Flash (#53906) with EPLB (#55119); K2-Horizon (#55063); Cohere Compass (#54774); Bailing V3 VL (#55921); Nanbeige4.2 via the Transformers backend (#56071); and a DeepSeek-V4 CPU backend with AVX512/AMX sparse MLA, indexer, mHC and compressor kernels (#55355).
- Fast Start: a persistent per-GPU weight-cache daemon holds post-quantized, TP-sharded weights in GPU memory so restarting engines map them over CUDA IPC with
--load-format ipc_cacheinstead of reloading from disk (#54921), now covering FP4 checkpoints (#55465) and multi-node TP (#55468). - Watermarking: Gumbel-max watermarked generation and detection with a keyed PRF, per-request opt-out and an example detection endpoint (#54053); dual-key Gumbel-max makes it compatible with speculative decoding (#56122); the Rust frontend forwards the per-request controls (#56338).
- HiSparse: a host-resident tier for sparse-MLA decode that spills KV pages to pinned host memory under GPU pressure and serves top-k misses from a per-request GPU hot buffer, enabled through
HiSparseConnector(#53781), with Prometheus counters (#56061), a host cache shared across TP ranks (#56629), and the attention config inferred from the connector (#57041). - Model Runner V2: dual-batch overlap in eager mode (#50945) and with FULL CUDA graphs for microbatched steps (#51700); MTP (#46994) and EAGLE3/DFlash/DSpark (#50514) speculative decoding under pipeline parallelism; adaptive verification for every draft-model speculator through an online acceptance estimator (#52228); gc frozen during graph capture, cutting capture from 12s to 2s and engine init from 28.9s to 8.2s on H200 (#54646);
--return-sampling-maskcompacted on GPU, fixing an about 2x RL step-time regression (#54901). - Qwen3.8-Flash-Next performance: separate prefill and decode QSA indexer kernels (#54513), fused PLE kernels (#54517), FP8 indexer cache (#54890), padded-index skipping in sparse GQA (#54873), fused PLE residual and QSA output gate (#55309), UVA PLE offload and Engram tensor parallelism via
--engram-config(#54371), and torch.compile removed from the NVIDIA implementation so FP8 fits on a single GB300 (#55272). - Kimi K3 performance: native CUDA AttnRes default on SM100 (#54261), KDA mixed-batch gather/scatter removed (5.2-7.7% E2E throughput, #56159), grouped FP8 MLA cache insertion (4-6x kernel speedup at small batch, #55356), DSV3 low-latency GEMM on strided tensors (12-81% kernel speedup, #54565), overlapped TP8 KDA projections (#54697), FlashInfer KDA kernels (#55364), internal prefix checkpoints with partial prefix caching and speculative decoding (#53614), and symmetric DCP disaggregation for hybrid Mamba models (#55531).
- Large scale serving: PCP+DCP on sparse-MLA models (#56157), PCP with single-module MTP and replicated DSpark (#56107) and decode-only FULL CUDA graphs (#53867), Elastic EP reusing CUDA graphs across reconfiguration (#54985), an opt-in FlashInfer PCIe IPC all-reduce for NVLink-less boxes (#53576), DeepEP v2 async finalize overlapping shared experts with combine (#52781), Mooncake Store heterogeneous TP sharing (#53129), a KVCR secondary-tier adapter (#53624), and encoder-cache sharing over NIXL (#47941) and Mooncake (#41567).
- Quantization: targeted online quantization through
quantization_config.targets(#51285) and on partially pre-quantized checkpoints from any quant method (#51392), W4A16 DSA with thenvfp4_fp8_ds_mlaKV cache (#51724), FlashInfer CuTeDSL NVFP4 W4A16 default over Marlin on SM100/103 (#53014), NVFP4 in the torch linear backend (#53319), per-quantization linear backend overrides (#51204), AutoRound 2/3/5/6/7-bit on CUDA (#52890), and DeepSelect top-k for the DSA sparse indexer (#56464). - Breaking changes: scale-out endpoints are opt-in on plain
vllm servevia--enable-scale-out, replacingVLLM_ENABLE_SCALE_OUT_ENDPOINTS(#54579, #55176); GPTQ activation ordering (g_idx) removed (#54809); items deprecated for 0.29 removed, including theVLLM_PREFIX_CACHE_RETENTION_INTERVALandVLLM_MM_HASHER_ALGORITHMenv vars (#55353); theallMamba cache mode deprecated (#55041);python -m vllm.entrypoints.grpc_serverdeprecated in favor ofvllm serve --grpc(#56746); YaRN aligned with Transformers so vendor YaRN aliases no longer re-scalemax_model_len(#56446).
Release Artifacts
Python Wheels
| Platform | Install |
|---|---|
| PyPI (CUDA 13.0) | pip install vllm |
| PyPI (CUDA 13.0, uv) | uv pip install vllm --torch-backend=auto |
| ROCm | pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.30.0/rocm723 |
| XPU | uv pip install vllm --extra-index-url https://wheels.vllm.ai/0.30.0/xpu --extra-index-url https://download.pytorch.org/whl/xpu --index-strategy unsafe-best-match |
Docker Images
| Platform | Docker Image |
|---|---|
| CUDA 13.0 (Default) | docker pull vllm/vllm-openai:v0.30.0 |
| CUDA 12.9 | docker pull vllm/vllm-openai:v0.30.0-cu129 |
| CUDA 13.0 + Ubuntu 24.04 | docker pull vllm/vllm-openai:v0.30.0-ubuntu2404 |
| CUDA 12.9 + Ubuntu 24.04 | docker pull vllm/vllm-openai:v0.30.0-cu129-ubuntu2404 |
| ROCm | docker pull vllm/vllm-openai-rocm:v0.30.0 |
| CPU | docker pull vllm/vllm-openai-cpu:v0.30.0 |
| XPU | docker pull vllm/vllm-openai-xpu:v0.30.0 |
Other Artifacts
Pre-built release artifacts are available in the Assets section at the bottom of this page, including:
- Source distribution tarball
- CUDA 12.9 Python wheels for x86_64 and arm64
- CUDA 13.0 Python wheels for x86_64 and arm64
- CPU Python wheels for x86_64, arm64, and macOS
- XPU Python wheel for x86_64
Model Support
- New models: DeepSeek-V4.1-Flash (#56214, #56228, #56208), DeepSeek-V4-Flash-Vision-Exp (#54566) with ROCm support (#55107) and LoRA (#55897), GLM-5.3-Flash (#53906), K2-Horizon with its reasoning and tool parsers (#55063), Cohere Compass (#54774, #55949), Bailing V3 VL with MTP (#55921), Nanbeige4.2 via the Transformers backend (#56071), and a DeepSeek-V4 CPU backend (#55355).
- DeepSeek-V4.1-Flash: whole KV in MXFP8 through the FlashMLA V4.1 record on SM100 (#56893), Mega-mHC from the DeepGEMM fork (#56962), mHC post block folded into the delayed pre projection (#56633), Triton-fused input metadata preparation (#56562), async prefetch for CPU-offloaded Engram lookups and Engram DP sharding (#56512), DSpark draft states collapsed before the SP all-gather (#56903), DSpark drafter no longer inheriting uninitialized EPLB state (#56387), XGrammar V4.1 schema constraints for strict tool parameters (#56408), Responses text parts (#56299), and image sentinel padding scoped to the Vision-Exp checkpoint (#56554).
- DeepSeek V4: plain RoPE without YaRN on sparse SWA layers (#54815),
--kv-cache-dtype autoresolving tofp8_ds_mlawith FlashMLA (#45091), prefill sparse index workspace sentinel seeded (#55299), dequant gather grid sized by rows (#55061), optional Q-norm in the fused MLA epilogue and group_size=32 packed FP8 quant (#56215), sparse settings read from the text config for composite models (#56160), MegaMoE startup without EP fixed (#55914), and a Triton iHC pre/post fallback for HY V4 (#55059). - GLM-5.3-Flash: EPLB support (#55119), FlashKDA for KDA chunked prefill (1.7-3.8x faster than the Triton chunk path, #55737), dense and masked-MHA sparse prefill for the NoPE (256, 0, 256) layout (#55738), decode hot-path cleanups removing per-layer copies and a duplicate router GEMM (#55736), and the
glm5nextNVIDIA subtree now packaged in wheels (#55214). - Qwen3.8-Flash-Next: separate prefill and decode QSA indexer kernels with
VLLM_SPARSE_INDEXER_MAX_LOGITS_MB(#54513), fused PLE kernels and merged K/V projections (#54517), sparse GQA skipping padded indices (#54873), FP8 indexer cache (#54890), fused PLE residual and QSA output gate (#55309), UVA PLE offload and Engram tensor parallelism via--engram-config(#54371), torch.compile removed from the NVIDIA path (#55272), compact indexer logits workspace (#54915), HC combine-norm reused for MTP input (#54687), PLE MTP metadata transfers (#55054), Hopper LL-GEMM tuning table (#54560), B200 FP8 MoE tuning (#55890), and fixes for FP8 PLE weight scales (#54722, #54882) and fused PLE conv strides (#55375). - Kimi K3: native CUDA AttnRes default on SM100 (#54261), ll_bf16 router weight prefetch (#53524), overlapped TP8 KDA projections (#54697), strided DSV3 GEMM (#54565), MLA decode epilogue latency (#54896), NVFP4 in_proj alignment routing to DeepGEMM (#55242), mixed-batch KDA without gather/scatter (#56159), grouped FP8 MLA cache insertion (#55356), FlashInfer KDA kernels (#55364), FlashKDA bf16 checkpoint state (#56485) and a stability bump (#54859), SiTU on the FlashInfer CuTeDSL MoE backend (#54606), internal prefix checkpoints with partial prefix caching and spec decoding (#53614), and fixes for RecoverSSM startup (#54634, #55774), GDN metadata (#54636), mixed-batch masks (#54781), checkpoint indexing (#55747, #56794), KDA illegal memory access (#55924), interleaved weight streams (#53379), NVFP4 MoE conversion OOM (#55407), and KDA projection overlap on Hopper (#55426).
- Transformers backend: attention found with a fuser and vLLM's layer attached to it (#54941), generic merged-column linear fusion (#55301), QKV fuser for Gemma4 (#55690), all vocab embeddings replaced including Gemma 3n/4 per-layer tables (#54760, #55083), and attention sinks applied (#52156).
- Speculative decoding for models: EAGLE3 for Sarvam MLA (#53052), Nemotron-H MTP with a separate, possibly quantized lm_head (#54574), NVFP4 DSpark gathered top-k projection for quantized Markov heads (#55713), Qwen3.5/3.6 multimodal MTP resolving
n_predictfrom the text config (#55369), Qwen3 DSpark padded-vocab drafts (#55133), and GLM-OCR MTP weight loading (#49869) and position masking under CUDA graphs (#56447). - Multimodal: torchcodec audio decoding and a selectable
audio_backendin--media-io-kwargs(#51826) with automatic decoding kept soundfile-first (#55642), torchaudio as the default resampler (#52598),media_io_kwargsin multimodal hashes (#54241), cache hash kwargs scoped by modality (#54918), empty video URLs with multimodal UUIDs (#54220), cached audio inputs with UUIDs (#56310), encoder CUDA graphs for MiniCPM-V 2.5/2.6/4.0 (#42785), Voxtral Realtime withFULL_DECODE_ONLYgraphs (#51167), Triton/FlashInfer composite attention for multimodal prefixes (#56305), pruned sliding-window tiles for Gemma 4 multimodal prompts (up to 3-4x E2E, #53147), SDPA for BLIP-2 Q-Former (#55285), baddbmm Conformer scores (#55062), fused DeepEncoder relative bias (#55629), Gemma3n sparse GELU Triton kernel (#48498), and no duplicate text embedding in Qwen2.5-Omni (#55415). - Correctness: Gemma 4 / Gemma 3n fine-tuned checkpoints without KV projections on shared layers (#54917, #55864), Gemma4 video preprocessing without image kwargs (#56652) and unified audio profiling (#56721), MiniCPM-V 4.6 loading (#49417) and video embeds (#54633), InternVL video parser (#55779), Nano-Nemotron precomputed embeddings (#56398), OpenPangu multimodal merge (#55941) and sleep mode (#53696), Qwen3-VL and Cosmos3-Edge under CPU and PP (#53699, #43272), Qwen VL processors without a tokenizer (#54886), Molmo2 fps guard (#55893), CohereASR audio-token estimate (#53829), double BOS in
LLM.chat()for multimodal models (#55288), decoded frame lists parsed as one video (#55326), swapped H/W in dummy video profiling (#56385), base64 video validation (#54323), Sentence Transformers 5.4+ configs (#54533), ColQwen3.5 pooler projector (#54847), torch.compile enabled for StableLM (#54969) and Sarvam MLA (#55817), and Nemotron latent-MoE skipping a redundant all-reduce at TP>1 (about 13% decode win, #52301). - LoRA: rsLoRA scaling in MoE expert packing (#55548), a log line when an adapter applies no weights (#55310), and
--lora-modules name=pathwith=in the path (#53353).
Engine Core
- Model Runner V2: DBO in eager mode (#50945) and with FULL CUDA graphs (#51700), MTP (#46994) and EAGLE3-style drafts (#50514) under pipeline parallelism, adaptive verification for all draft-model speculators (#52228), gc frozen during capture (#54646), GPU-compacted sampling masks (#54901), UVA-backed
apply_write(#55819), fast-prefill support (#56145), DP sync skipped for all speculator uniform decodes (#54856), prompt embeddings in decoder pooling (#56639), pooling post-processing on non-final PP ranks (#56666), kernels warmed before graph capture (#55341),CUDAGraphStat(#52358), Proton CUDA graph attribution (#51084), platforms without pinned memory such as WSL (#56908), draftmoe_backend(#54788) andattention_backend(#54826) overrides honored, DCP metadata initialized after batch partitioning (#55212), and arecord_streamfix in the PP draft broadcast (#55745). The MRV2-specific Triton JIT warmup migration was reverted (#56654). - Speculative decoding: opt-in
disable_eagle_block_drop(#53388), adaptive verification onFLASHINFER_MLA_SPARSE_DSV4(#52724) and deferred until after kernel warmup (#55455), full-vocab EAGLE3 drafters skipping vocab remapping (#53458), DSpark preserving the target DCP config (#55472), DFlash drafters dropping FlashAttention's AOT schedule (#54374), DP token padding excluded from draft metadata (#55458, #56181), fastsafetensors deadlock avoided for PP drafts (#54416), Mamba state cached at the EAGLE resume position (#53945) and preserved for padded prompt tails (#55178), resumed requests padded in P/D (#55126), and warmup device selection (#55245). - Prefix caching and scheduling: both replay boundaries retained so an EAGLE resend of a block-aligned prompt still hits (#54713), zero-progress preemption cascades stopped for deferred KV frees (#49675), SWA layers taking the primary block size to avoid inflating the KV block LCM (#53007), align-mode Mamba states retired across null gaps (#55450), the
allMamba cache mode deprecated with MRV1 fallback (#55041), andvllm:request_num_preemptionshistogram (#49984). - Kernels: FlashInfer ReplaySSM backend for Mamba2 (#52506), B12X causal paged attention for SM120/SM121 via
--attention-backend B12X_ATTN(#52017), FlashInfer XQA decode on SM90 (#50439), DeepSelect top-k with--sparse-indexer-topk-backend(#56464), BF16x3 router GEMM default on SM100 for FP32-router models (#55899), split-row Triton top-p for small batches (#54651), manualActivationQuantFusionPassproducer for Llama (#51415), fused add-RMSNorm + NVFP4 (#51925), prefix-prefill tiling independent of KV page size (#54194), FlashInfer GDN prefill on SM12x (#55715), fused GDN MTP decode for SM110 (#53835), masked MoE activation for padded layouts (#51217), cooperativebatched_moe_align_block_size(#53280), PDL infusedQKNormRopeKernel(#55755) and weight prefetch in fused q/kv RMSNorm (#55020), XD-RoPE folded into M-RoPE (#56078), and the FlashAttention pin synced with upstream (#54819). - Startup and warmup: renderer warmup overlapped with engine core init (#54557), FlashInfer autotune no longer rerun on each source change (#54794), Qwen3.5/Qwen3-Next Triton kernels (#54797) and GDN gated RMSNorm (#54251) warmed at startup, Gemma 4 de-JITified (#55768), DeepSeek V4 and shared attention, sampling, DFlash and mHC kernels migrated to the JIT warmup registry (#50175, #53564, #53565, #50176, #53566, #56323, #50178) with parallel mHC pre-norm compilation (#56683), FlashInfer deferred MoE decode kernels autotuned before capture (#55377), and supplemental BF16 autotuning isolated (#57285).
- Fast Start: weight-cache daemon with
--load-format ipc_cache(#54921), FP4 checkpoints (#55465), multi-node TP (#55468), GPU UUID socket folders (#56669), per-client IPC tensor export so a copy-mode client cannot release zero-copy weights (#56472), and a clear error on non-CUDA/ROCm platforms (#56010). - Watermarking: Gumbel-max watermarked generation and detection (#54053) and dual-key Gumbel-max for speculative decoding with
allow_target_only_watermarkingand context deduplication (#56122). - Sleep and RL: NCCL communicator memory released in sleep mode (#51485), DP state synced on the first step of a wave so pause and sleep stop burning 32 dummy forwards per rank (#52957), pre-shared
ncclUniqueIdrendezvous for torch-free trainers (#53784), stale HPC QK-norm weights after refit fixed (#56706), andsession_idon GPUBlockStoredevents (#51381). - Robustness: attention implementations must explicitly declare DCP support (#55780), unavailable piecewise CUDA graphs raise instead of producing garbled output (#54782), non-compiled models fall back to
FULL_DECODE_ONLYgraphs (#55095), MRV1 breakable graphs scoped to the piecewise path (#56312), sync checks cover paged async copies (#53491) and torch.compile (#56904), more H2D copies pinned (#54660, #55202, #56138), GDN capture metadata built without a device sync (#55404), queued work carried over when materializing the dedicated stream (#56382), FlexAttention recompiles avoided (#55888), persistent top-k falling back on low-shared-memory GPUs (#54110), KV page size scaled for hidden-state extraction with TP (#50894), out-of-bounds access in SM90 sparse MLA mixed batches (#56969), sparse-MLA piecewise capture crash (#56825), TRT-LLM ragged prefill regression (#55499), MoE fused sum int32 overflow (#50220), odd-row per-token-group quant cliff (#56478), dummy-load re-initialization (#56682, #56688),warning_onceretaining the LLM instance (#54098), DSpark cache-group capability underpython -O(#55234), sparse-MLA DCP stabilized for GLM PCP configurations (#55879), DeepGEMM FP8 warmup coverage (#56452), Triton autotune inspection skipped without Triton (#56676),mm_device_do_normalizeunder encoder CUDA graphs (#55370), VidCom2 frame budgets read once (#55331), the CUDA profiler start fix (#55237), FlashInfer all-reduce workspace failures logged as errors (#55127), and backend-neutral KV cache and MFU log lines (#55650).
Hardware & Performance
- NVIDIA: FlashInfer PCIe IPC all-reduce (#53576), DeepGEMM pinned to the vLLM fork 2.8.0 with SM120 and SM90 paged-MQA ports (#56876), public CUDA 13.4 Rubin build path (#54640, #56545), SM12x blockwise FP8 CTA raster swizzle for GB10/DGX Spark (#55180), W4A4 NVFP4 preferred on SM120/121 (#55170), H20 block-FP8 MoE tuning (+21%, #54668), an A100 PCIe fused-MoE config for Qwen3.5-122B TP2 (#55511), ReLU2 fused with static FP8 quant for Nemotron-H (#53793), and NVFP4 KV cache FMHA sped up (#55031).
- AMD ROCm: TheRock base bumped to ROCm 10.0 (#55246) with nightly TheRock images (#55014), AITER 0.1.21.post2 (#52826, #55968), AITER custom AG/RS on the DP group (about 3% TPOT, #48247), multi-stream performance and rocprofiler fixes (#55099, #56098), bpreshuffled blockscaled FP8 GEMM (+4-8% QPS on DSv3, #51692), low-M FP32 router GEMM on gfx950 (#54845), DeepSeek V4 FHMoE shared-expert fusion (#53161), CSA multi-stream overlap (#51794), AITER OPUS for large sparse prefill (#54855), DSv4.1 mHC through AITER (#56503, #56513), MXFP8 dequant-once (#56560), strided DSA decode mask (#56628) and K=512 top-k tuning (#56743), MiniMax-M3 indexer and top-k work (#54682, #52664, #55235, #56170) with fused allreduce+GemmaRMSNorm (#54787), Kimi-K3 fused KDA prefill relanded (#54038) and fused MLA norms (#52494), GLM-5.2 kept on MRV1 (#53155), GLM-5.3-Flash MTP through ragged sparse MLA (#55239), W4A16 packed zero-points (-26% TPOT on Gemma 4 AWQ, #54965), Triton W4A16 GEMM as a custom op (#51453), split AITER CK/Triton MXFP4 backends (#50622, #54773), RDNA3 W4A16 MoE on the oracle path (#55522), Quark per-block FP8 MoE (#52263), KV connectors with
ROCM_AITER_UNIFIED_ATTN(#53695), MoRI-IO SWA+HMA (#53721), attention sinks in AITER sparse MLA (#54404), non-causal DSpark drafts in AITER MLA decode (#55966), shared KV prefill (#55887), V-cache padding masked (#53856), elastic EP deadlock (#56610), profiler fixes (#54171, #56190), breakable CUDA graphs auto-enabled for DeepSeek-V4.1 (#56349), the AITER paged-MQA outputs guard removed for DeepSeek-V4 (#55808), DSV4 indexer gather cold-compile churn removed (#56153), PDL JIT warmup skipped where unsupported (#56035), Kimi-K3 fused prefill state_indices crash (#56526), Ray NIXL agents for sharded RDT (#56687), Qwen3-Omni audio encoder TP with non-divisible heads (#45900), Quark MXFP4 Qwen3.8-Flash-Next shared experts on gfx950 (#55213), AITER preshuffled FP8 block-scale kernel (#56433), linear biases withrequires_grad=False(#56161), a clearer unsupported KV dtype error (#55236), Mooncake in the ROCm image (#55002), and AINIC pins (#54112, #56459). - Intel XPU: activation CustomOps on SYCL kernels (#53734), batch-invariant matmul and linear kernels (#49209), fused GemmaRMSNorm (#53678),
fused_input_norm(#52945) and fusedgrouped_topk(#53580), LoRA for DeepSeek V4 (#53689), DP external LB device assignment (#53037), triton-xpu 3.8.0 shim (#56014), auto-round-lib 0.15.0 (#54515), and fixes for FalconH1 PP (#55913), Ernie4.5-VL RoPE (#55942), gated RMSNorm ops (#54968), Qwen DFlash context keys (#56431), INC quant selection (#47237), andmoe_wna16loading (#52651). - CPU: DeepSeek-V4 backend (#55355), native AMX-FP8 attention for Diamond Rapids with the Docker base moved to Ubuntu 25.04 and GCC 15 (#49410), faster LM head on Arm (about 6% on Llama-3.1-8B W8A8, #55352), DeepSeek-R1 FP8 MLA + MoE correctness (#56773), profiler tables grouped by input shape (#56016), a dedicated Zen5 image (#50314), and nightly CPU images on Docker Hub (#55163).
Large Scale Serving
- Context parallelism: PCP+DCP on sparse-MLA models (#56157), PCP with single-module MTP and replicated DSpark (#56107), PCP decode-only FULL CUDA graphs (#53867), FlashMLASparse MTP at CP interleave > 1 (#56722), symmetric DCP disaggregation for hybrid Mamba models (#55531), DCP empty-shard LSE mask fused into the A2A pack kernel (#54889), DCP prefill keys materialized on non-owner ranks (#54908), DCP indexer interleave guard removed (#55802) and respected in the PCP gather (#56715), PCP producer shards as NIXL transfer ranks (#56645), replicated-PCP ranks reporting done (#53903), the DCP interleave adjustment scoped to NixlConnector (#54803), PCP counted in multi-node world size (#55111), PCP-DP validation moved to the GPU manager (#54523), and one-sided EP at DP=1 (#54016).
- Elastic EP and MoE communication: CUDA graphs reused across reconfiguration (#54985), routing only to surviving engines on scale-down (#55772), DeepEP v2 async finalize (#52781), FlashInfer PCIe IPC all-reduce (#53576, #54869), batched CUTLASS workspace sizing (#55579), and PP never dropping a decoding request from the sampled-token broadcast (#54436).
- KV connectors: Mooncake Store heterogeneous TP sharing (#53129), heterogeneous PP completion (#56033), replicated GQA heads (#52516), physical-block transfer length (#54272), producer partial Mamba tails (#52832), hybrid-model finish-time saves (#54643), failed remote loads reported to the scheduler (#50984), QSA ring groups excluded (#55027); NIXL per-region transfer geometry (#53780), int32 indices (#51952), multi-handle race (#56104), no eviction mid-transfer (#54689), full prefix hits reported finished (#56640), notification-only requests (#57570), and double cleanup (#54518); DecodeBench HMA, DCP, prefix and circular-buffer fixes (#54647, #54679, #54878, #54879); connector block tables resolved for every scheduled request (#54853), hybrid KV load failures (#50388), and
lmcache_mp_connectorstate transitions guarded (#47505); HiSparse per-layer offsets (#57027), P/D region-mapped pulls (#57077) and NIXL full-block imports (#57049); NixlPush remote-agent reads guarded (#56317). - KV offloading: KVCR secondary-tier adapter (#53624), P2P tier
unbound_store_timeout_sand late-fetch rejection (#53453), OffloadingConnector retention interval (#51886) and offload hits under MTP/EAGLE (#52771), SimpleCPUOffload fine-grained hybrid prefix hits (#54736), eager registration and final flush (#53532),BlockStoredmetadata (#54325), mixed page sizes for DSA models (#54756), prefix-cache bypass (#54998), no-forward-step stores (#56621); UniformTypeKVCacheSpecs portability (#51690) and DCP scaling (#50883); and fixes for async lookups (#54872, #55075, #55823), SWA reachability and coverage (#54362, #55712, #56486), final-token slot (#54288), oversized offers (#54759), ARC fallback (#55461), disk alignment (#55424), DiskBackend buffer race (#51667), offload key ordering (#52923), prefetch slot ownership (#54975), load boundaries (#52807), and tiering shutdown (#52290). - Encoder cache and EPD: P2P NIXL + CPU EC connector (#47941), ECMooncakeConnector (#41567), embedding inputs auto-enabled on EC/KV consumers (#56379), GPU NVDEC for encoder-only instances (#53675), lightweight multimodal metadata from render (#54659) as JSON arrays (#56090), reduced proxy serialization overhead (#56657), media options and UUIDs preserved (#56786, #56432), per-item EC checks (#56070), ECCPUWorker backend extension points (#54033), and fixes for old kernels (#53190), remote encoding failures (#55290), and TP>1 load devices (#40416).
Quantization
- New capabilities: targeted online quantization (#51285) and online quantization of partially pre-quantized checkpoints (#51392), W4A16 DSA (#51724, #55538), FlashInfer CuTeDSL NVFP4 W4A16 (#53014), NVFP4 torch backend (#53319),
linear_backend_per_quant(#51204), AutoRound 2/3/5/6/7-bit (#52890), TRT-LLM FP8 block-scale MoE with SwiGLU clamp (#55069), and Quark per-block FP8 MoE on ROCm (#52263). - Refactors: Quark dispatch on
QuantKey(#52958) and ModelOpt linear methods unified into one QuantKey-driven method (#49381). - Fixes: NVFP4 padding initialized in-kernel (#53568), unloaded NVFP4 scales detected with a NaN sentinel (#52501), NVFP4 fused SiLU+mul scales (#55643), Quark per-block scales registered as
weight_scale(#54770), GPT-OSS FP8 MoE with Quark (#54824), FSE detection for Quark models (#54573), DSv4 MXFP4 backend alias selection (#53586), unsetgroup_sizeon compressed-tensors WNA16 MoE (#53163), block FP8 MTP in ModelOpt mixed checkpoints (#55513), and unquantized MoE weight storage preserved on ROCm for RL updates (#46009).
API & Frontend
- New endpoints and options: stateless
/v1/responses/render(#50195), scale-out endpoints behind--enable-scale-out(#54579, #55176),count_reasoning_tokensso usage reportsreasoning_tokens(#54982), prompt metadata from/inference/v1/generate(#53187),vllm:request_num_preemptions(#49984), MFU/MBU for SWA and hybrid layers (#55624), consolidated Prometheus bucket defaults (#48866), site-packages reasoning and tool parser plugins (#45241), andmax_num_queued_reqsshared across API server processes (#54746). - Structured output and parsers: XGrammar
patternProperties/propertyNames/unevaluatedProperties(#42904), full-history reasoning scans eliminated (#55223) withis_reasoning_endderived from the grammar (#56200), invalid structured-output requests no longer stopping the engine (#51450), a unified Cohere Command parser (#56392), MCP SDK 2.x schemas (#53870), incomplete tool-call markup dropped consistently (#47562), DSML tool calls without the wrapper (#55954), misspelled wrappers (#56141) and implicit parameter closes (#54838), Gemma4 malformed openers (#53444), Seed-OSS turn boundaries (#54264), Step-3.5 with structured outputs (#53174), reasoning parser plugins loaded before headless config (#53124), and a warning for removedguided_*fields (#54285). - OpenAI, Anthropic and Cohere compatibility: unsupported Responses input items return 400 (#55974),
browser.findaction type (#55305), Responses validation boundary restored (#55761), validation errors migrated toVLLMValidationError(#50254, #50257, #55701, #55735), unsupportedreasoning_effortas a client error (#54022), content format detected through macro parameters (#53824), Anthropicstop_sequenceinmessage_delta(#55325), Cohere priorities bounded (#56146), translation rejecting string files (#55618), tokenless chat and audio streams rejected (#54708), beam search respectingskip_special_tokens(#56211) and aborts (#56249), nested cancellation handlers removed (#55602, #57024), extension integers validated (#55606), routed-expert prompt offsets validated (#56844), MistralCommonBackend avoided for HF tokenizers (#54192), chat template fallbacks packaged (#53762), and the content format warning restored (#54622). - Render and pooling:
assistant_tokens_mask(#54539) and token offsets (#54692) kept aligned with truncated prompts,skip_decoder_start_tokenhonored (#54799), renderer warmup bounded (#55448), LoRA adapter names in pooling responses (#56004),max_embed_lenfor chunked embeddings (#55551), bodyrequest_idhonored (#55665), actual input usage for scoring (#56573), STEP token pooling (#55307), offline Jina token limits (#56415), and a warning for Qwen3 rerankers without a chat template (#56017). - Rust frontend: TLS for the render server (#54999),
--lora-modules(#54837), HTTP RL weight synchronization (#56567), HF revisions, offline mode and cache directory (#56386),{% generation %}chat template blocks (#56378), thevllm-protocrate on crates.io (#56365),reasoning_tokensin usage (#54883) built on token-attributed text (#52910, #54884), Mooncake/NIXL connector metrics (#52755) andvllm:iteration_tokens_total(#56990), gRPC preprocessed multimodal features (#55047), remote-prefill multimodal metadata (#54814) and engine errors surfaced as INTERNAL (#56405), Qwen4-exp multimodal (#54813), optional--max-model-lenfor render (#55328), per-request vocabulary counting removed (#54449), decoded chunks coalesced per update (#55012), bounded recursive parsers (#54303), tekken 0.2.0 (#53056), strict whitespace framing (#55417), consistent parser resolution (#56018), empty structured-output values rejected (#54821), selected-token logprobs (#56406), undefined token ids skipped (#55240), DeepSeek renderer fixes (#53281, #54854, #56593, #56260, #56366), and HF ShareGPT multi-turn datasets invllm bench serve(#51104, #56915). - CLI and benchmarks:
vllm bench serveaccounts for client queue time (#54136) with consistent streaming TTFT/E2E accounting (#55508), non-positive--max-concurrencyrejected (#54887),mm-processorcrash fixed (#56300), complete pooling responses measured (#56760), explicit empty and zero arguments honored invllm chat/complete(#55710),vllm launch rendershutdown hang fixed (#54913), double/stop_profilecrash fixed (#51678), and remaining non-OpenAI content moved out of the openai folder (#56369).
Security
- Validation-error response bodies bounded, closing an about 5,300x response amplification (#54684); client-supplied sparse embeddings bounded before densification (#54632).
- Request-controlled video sampling capped for GLMGA (#54935) and Qwen-VL (#56729) backends.
cache_saltvalidated before reaching LMCache so one request cannot take down the engine (#51444); scale-out multimodal features validated before engine handoff (#51898); late-interaction query caches keyed by server-generated ids instead ofX-Request-Id(#51445).- Credentials redacted from benchmark logs (#56662); Rust frontend HTTP method labels normalized to prevent unbounded Prometheus series (#56058).
Dependencies
- CUTLASS 4.7.1 (#54190) and CUTLASS DSL 4.7 (#54927), Transformers 5.16.1 (#53905), tpu-inference v0.29.0 (#56388),
openai >= 2.25.0(#49104),huggingface_hub >= 1.31.0with httpx imported through it (#56460). - DeepGEMM pinned to the vllm-project fork 2.8.0 (#56876) with CUDA 12.9 release builds fixed (#57554); FlashMLA V4.1 record pin (#56893); FlashKDA bumps (#54859, #56485); FlashAttention synced with upstream (#54819);
_USE_MATH_DEFINESdefined for FlashMLA targets (#54007). - Public CUDA 13.4 Rubin build path and dependency pins (#54640, #56545).
- ROCm: TheRock base on ROCm 10.0 (#55246), AITER 0.1.21.post2 (#52826, #55968), Mooncake from public wheels (#55002), AINIC libionic 54.0-187-1 pinned to a snapshot (#54112, #56459).
- XPU: triton-xpu 3.8.0 shim (#56014), auto-round-lib 0.15.0 (#54515).
- CPU: Docker base on Ubuntu 25.04 with GCC 15 (#49410), Zen5 image (#50314), nightly CPU images (#55163). Obsolete TPU Dockerfile removed (#55376).
Breaking Changes & Deprecations
- Scale-out endpoints (
/render,/derender,/inference/v1/generate) are no longer registered on plainvllm serveunless--enable-scale-outis passed;VLLM_ENABLE_SCALE_OUT_ENDPOINTSwas removed.vllm launch renderandvllm serve --tokens-onlyalways register them (#54579, #55176). - GPTQ group/dynamic activation ordering removed:
g_idxis ignored and the related Marlin, GPTQ, CPU and RDNA3 kernels are gone (#54809). - Items deprecated for 0.29 removed:
VLLM_PREFIX_CACHE_RETENTION_INTERVALandVLLM_MM_HASHER_ALGORITHMenv vars (use the config fields), theuse_fp4_indexer_cachealias (useindexer_kv_dtype), theCUDA_VISIBLE_DEVICESfallback on ROCm (useHIP_VISIBLE_DEVICES), and theseq_lens_cpu/num_computed_tokens_cpuattention metadata properties (#55353). - The
allMamba cache mode is deprecated and falls back to Model Runner V1 (#55041). python -m vllm.entrypoints.grpc_serveris deprecated; usevllm serve --grpc(#56746).sse_keep_alivemoved tovllm.entrypoints.serve.utils(#56369).- YaRN aligned with Transformers: vendor YaRN aliases no longer multiply
max_position_embeddingsby the factor a second time, so derivedmax_model_lendrops for some models (TeleChat3-36B-Thinking 131072 to 32768, sarvam-105b 5242880 to 131072);attn_factorandextrapolation_factorare ignored in favor ofmscale/attention_factor(#56446). - Attention implementations must explicitly declare DCP support; DCP with ROCm standard attention, Triton, FlexAttention or TurboQuant now fails at backend selection (#55780).
- MoRI-IO connector WRITE mode with hybrid KV cache groups now requires
--disable-hybrid-kv-cache-manager; use READ mode otherwise (#53721). - Default audio resampler switched from PyAV to torchaudio (#52598); audio decoding is backend-selectable with soundfile first in
auto(#51826, #55642). - New defaults: FlashInfer CuTeDSL NVFP4 W4A16 over Marlin on SM100/103 (#53014), W4A4 NVFP4 over weight-only kernels on SM120/121 (#55170), BF16x3 router GEMM on SM100 (#55899), DeepEP v2 combine overlap on with
VLLM_DEEPEP_V2_COMBINE_OVERLAP=0to disable (#52781), AITER custom AG/RS on for DP attention + TP experts withVLLM_ROCM_USE_AITER_CUSTOM_AR=0to disable (#48247), native CUDA AttnRes for Kimi-K3 on SM100 (#54261), andVLLM_USE_BREAKABLE_CUDAGRAPH=1no longer overriding FULL graphs on MRV1 (#56312).
New Contributors
- @100milliongold made their first contribution in https://github.com/vllm-project/vllm/pull/56017
- @404mario made their first contribution in https://github.com/vllm-project/vllm/pull/52781
- @982945902 made their first contribution in https://github.com/vllm-project/vllm/pull/52923
- @abrahamzewoudie made their first contribution in https://github.com/vllm-project/vllm/pull/45900
- @aijanai made their first contribution in https://github.com/vllm-project/vllm/pull/51678
- @Aj2280 made their first contribution in https://github.com/vllm-project/vllm/pull/55728
- @Akine-Ko made their first contribution in https://github.com/vllm-project/vllm/pull/54033
- @akshaver made their first contribution in https://github.com/vllm-project/vllm/pull/53945
- @alexliluz made their first contribution in https://github.com/vllm-project/vllm/pull/53124
- @am-cohere made their first contribution in https://github.com/vllm-project/vllm/pull/54774
- @amd-xavierwang made their first contribution in https://github.com/vllm-project/vllm/pull/56153
- @andy-neuma made their first contribution in https://github.com/vllm-project/vllm/pull/50314
- @AnshulDesai made their first contribution in https://github.com/vllm-project/vllm/pull/55271
- @ArcheyChen made their first contribution in https://github.com/vllm-project/vllm/pull/54436
- @arcusbuilds made their first contribution in https://github.com/vllm-project/vllm/pull/54007
- @averma12 made their first contribution in https://github.com/vllm-project/vllm/pull/56721
- @bakiburakogun made their first contribution in https://github.com/vllm-project/vllm/pull/55511
- @BANANASJIM made their first contribution in https://github.com/vllm-project/vllm/pull/55301
- @Big2Wheel made their first contribution in https://github.com/vllm-project/vllm/pull/54975
- @bjf-frz made their first contribution in https://github.com/vllm-project/vllm/pull/55819
- @bojiang3 made their first contribution in https://github.com/vllm-project/vllm/pull/54306
- @BolinSNLHM made their first contribution in https://github.com/vllm-project/vllm/pull/54606
- @BruceLoveDecimal made their first contribution in https://github.com/vllm-project/vllm/pull/50984
- @byshiue made their first contribution in https://github.com/vllm-project/vllm/pull/55054
- @cenab made their first contribution in https://github.com/vllm-project/vllm/pull/55710
- @circlepen made their first contribution in https://github.com/vllm-project/vllm/pull/56388
- @colinmcnamara made their first contribution in https://github.com/vllm-project/vllm/pull/55325
- @DebugSy made their first contribution in https://github.com/vllm-project/vllm/pull/55111
- @dharak-cohere made their first contribution in https://github.com/vllm-project/vllm/pull/53784
- @djw8605 made their first contribution in https://github.com/vllm-project/vllm/pull/54962
- @eastwood-c made their first contribution in https://github.com/vllm-project/vllm/pull/55745
- @erdholion made their first contribution in https://github.com/vllm-project/vllm/pull/54285
- @gc-fu made their first contribution in https://github.com/vllm-project/vllm/pull/55310
- @happyyzy made their first contribution in https://github.com/vllm-project/vllm/pull/50220
- @heliubj18 made their first contribution in https://github.com/vllm-project/vllm/pull/54756
- @Ianniu123 made their first contribution in https://github.com/vllm-project/vllm/pull/55650
- @ItsRoy69 made their first contribution in https://github.com/vllm-project/vllm/pull/55283
- @jackLei0901 made their first contribution in https://github.com/vllm-project/vllm/pull/49869
- @Jankwi made their first contribution in https://github.com/vllm-project/vllm/pull/54220
- @JaynouOliver made their first contribution in https://github.com/vllm-project/vllm/pull/47562
- @jcayab made their first contribution in https://github.com/vllm-project/vllm/pull/54584
- @jorge-menjivar made their first contribution in https://github.com/vllm-project/vllm/pull/53056
- @josiahdavis made their first contribution in https://github.com/vllm-project/vllm/pull/56286
- @jschmied made their first contribution in https://github.com/vllm-project/vllm/pull/55180
- @jyizheng made their first contribution in https://github.com/vllm-project/vllm/pull/54518
- @kamb-code made their first contribution in https://github.com/vllm-project/vllm/pull/52771
- @krsish made their first contribution in https://github.com/vllm-project/vllm/pull/55307
- @kushaldabbe made their first contribution in https://github.com/vllm-project/vllm/pull/55548
- @LauraGPT made their first contribution in https://github.com/vllm-project/vllm/pull/54944
- @LearningMachine621 made their first contribution in https://github.com/vllm-project/vllm/pull/49675
- @linamy85 made their first contribution in https://github.com/vllm-project/vllm/pull/49984
- @lucamotz made their first contribution in https://github.com/vllm-project/vllm/pull/54110
- @luoyuctl made their first contribution in https://github.com/vllm-project/vllm/pull/56387
- @luyixiao95 made their first contribution in https://github.com/vllm-project/vllm/pull/52285
- @lvliang-intel made their first contribution in https://github.com/vllm-project/vllm/pull/47237
- @lzzzzzc made their first contribution in https://github.com/vllm-project/vllm/pull/55864
- @maireneu made their first contribution in https://github.com/vllm-project/vllm/pull/54533
- @mevince made their first contribution in https://github.com/vllm-project/vllm/pull/54325
- @migarci2 made their first contribution in https://github.com/vllm-project/vllm/pull/55602
- @mingg26 made their first contribution in https://github.com/vllm-project/vllm/pull/53524
- @miyakido made their first contribution in https://github.com/vllm-project/vllm/pull/40416
- @mlazos made their first contribution in https://github.com/vllm-project/vllm/pull/53319
- @mohit-sarvam made their first contribution in https://github.com/vllm-project/vllm/pull/53052
- @natsala13 made their first contribution in https://github.com/vllm-project/vllm/pull/55178
- @null-Exception1 made their first contribution in https://github.com/vllm-project/vllm/pull/54265
- @odashi made their first contribution in https://github.com/vllm-project/vllm/pull/45241
- @Oskii made their first contribution in https://github.com/vllm-project/vllm/pull/55618
- @pra2107tham made their first contribution in https://github.com/vllm-project/vllm/pull/54622
- @Roderick-Wu made their first contribution in https://github.com/vllm-project/vllm/pull/54809
- @SalikovAlex made their first contribution in https://github.com/vllm-project/vllm/pull/56414
- @seanwestfall made their first contribution in https://github.com/vllm-project/vllm/pull/55691
- @SherifWaly made their first contribution in https://github.com/vllm-project/vllm/pull/53379
- @somuai made their first contribution in https://github.com/vllm-project/vllm/pull/55369
- @specture724 made their first contribution in https://github.com/vllm-project/vllm/pull/50945
- @StevenWang-CY made their first contribution in https://github.com/vllm-project/vllm/pull/54819
- @stu-cao made their first contribution in https://github.com/vllm-project/vllm/pull/55802
- @subhashpolisetti made their first contribution in https://github.com/vllm-project/vllm/pull/54633
- @surajm20061998 made their first contribution in https://github.com/vllm-project/vllm/pull/55508
- @Sy0307 made their first contribution in https://github.com/vllm-project/vllm/pull/55212
- @tanyuqian made their first contribution in https://github.com/vllm-project/vllm/pull/55063
- @TensorRaya made their first contribution in https://github.com/vllm-project/vllm/pull/55908
- @Ubospica made their first contribution in https://github.com/vllm-project/vllm/pull/56408
- @wei-core made their first contribution in https://github.com/vllm-project/vllm/pull/53835
- @Whamp made their first contribution in https://github.com/vllm-project/vllm/pull/54362
- @Woolgathererer made their first contribution in https://github.com/vllm-project/vllm/pull/56652
- @wuhangxian made their first contribution in https://github.com/vllm-project/vllm/pull/53824
- @xiaozhenbi made their first contribution in https://github.com/vllm-project/vllm/pull/56671
- @xonder made their first contribution in https://github.com/vllm-project/vllm/pull/43272
- @xuhuan51 made their first contribution in https://github.com/vllm-project/vllm/pull/51381
- @YeonwooSung made their first contribution in https://github.com/vllm-project/vllm/pull/56593
- @ykamiset made their first contribution in https://github.com/vllm-project/vllm/pull/52664
- @YoavMiron made their first contribution in https://github.com/vllm-project/vllm/pull/54574
- @yongqinwang-cmd made their first contribution in https://github.com/vllm-project/vllm/pull/50514
- @Yuzu23 made their first contribution in https://github.com/vllm-project/vllm/pull/45091
- @yzeyu71 made their first contribution in https://github.com/vllm-project/vllm/pull/50504
- @z-zanez made their first contribution in https://github.com/vllm-project/vllm/pull/53129
- @zdtsw made their first contribution in https://github.com/vllm-project/vllm/pull/54999
- @zhang-keliang made their first contribution in https://github.com/vllm-project/vllm/pull/55288
- @zhaoguochun1995 made their first contribution in https://github.com/vllm-project/vllm/pull/56382
- @zhouyou9505 made their first contribution in https://github.com/vllm-project/vllm/pull/54659
- @zigzagcai made their first contribution in https://github.com/vllm-project/vllm/pull/54560
- @zqlcode made their first contribution in https://github.com/vllm-project/vllm/pull/56071
- @zvier made their first contribution in https://github.com/vllm-project/vllm/pull/56103
- @zwang86 made their first contribution in https://github.com/vllm-project/vllm/pull/54098
Contributors
@khluu, @AndreasKaratzas, @taneem-ibrahim, @BugenZhao, @yewentao256, @njhill, @NickLucche, @aoshen02, @LopezCastroRoberto, @WoosukKwon, @LucasWilkinson, @zyongye, @hmellor, @Fangzhou-Ai, @sfeng33, @Isotr0py, @gau-nernst, @JaredforReal, @wzhao18, @fxmarty-amd, @stefankoncarevic, @Alex-ai-future, @mgoin, @micah-wil, @mayuyuace, @djramic, @ZJY0516, @mawong-amd, @Rohan138, @sychen52, @waizuichougou, @chaunceyjiang, @simondanielsson, @KernelClint, @AdaAibaby, @gty111, @zhenwei-intel, @zxd1997066, @majunze2001, @divakar-amd, @zhewenl, @giuseppegrossi, @Levius-Fubuki, @gcanlin, @jperezdealgaba, @bigPYJ1151, @stecasta, @linitra24, @chaojun-zhang, @shaohuaxi, @reidliu41, @atalman, @mfylcek, @lucifer1004, @ZeldaHuang, @askliar, @peakcrosser7, @yzeyu71, @hungnnvidia, @zupengwang, @TheEpicDolphin, @hlin99, @liusy58, @jyizheng, @jimmy-adams, @lucamotz, @jeejeelee, @shen-shanshan, @MatthewBonanni, @cjackal, @aarushjain29, @noooop, @Juntian777, @FeathBow, @Hotragn, @Jankwi, @lzhan011, @subhashpolisetti, @maithilijoshi20, @ilmarkov, @lukealonso, @mingg26, @franciscojavierarceo, @mevince, @zdtsw, @bnellnm, @omerpaz95, @jikunshang, @rbrugaro-amd, @bohnstingl, @specture724, @afierka-intel, @mohit-sarvam, @andylolu2, @pisceskkk, @starkwj, @wangyicong52, @eastwood-c, @wuhangxian, @yzong-rh, @yma11, @TQCB, @Sy0307, @jbyczkow, @itayalroy, @xiaguan, @alec-flowers, @ayush1399, @orestis-z, @mobicham, @ArcheyChen, @Zhenzhong1, @ZhengGong-amd, @jorge-menjivar, @IwakuraRein, @linamy85, @wentian-byte, @afriedri, @dharak-cohere, @DarkLight1337, @lvliang-intel, @zigzagcai, @harjothkhara, @bojiang3, @maireneu, @z-zanez, @JaynouOliver, @pra2107tham, @jdebache, @aijanai, @Dao007forever, @juhi10071998, @odashi, @Etelis, @fjosw, @jackLei0901, @tarukumar, @positive666, @tzielinski-habana, @drakosha, @iyastreb, @Harry-Chen, @sheralskumar, @ccrhx4, @aaab8b, @982945902, @tanyuqian, @DebugSy, @jasonozuzu-cohere, @byshiue, @yifjiang, @BolinSNLHM, @wseaton, @Yuzu23, @andyxning, @andy-neuma, @foraxe, @xuhuan51, @AnshulDesai, @luyixiao95, @null-Exception1, @yiz-liu, @zhang-keliang, @erdholion, @happyyzy, @GirasoleY, @SubSir, @natsala13, @yongqinwang-cmd, @StevenWang-CY, @LauraGPT, @wei-core, @zhejiangxiaomai, @bakiburakogun, @Whamp, @almogtavor, @zhouyou9505, @jschmied, @wenhuach21, @Big2Wheel, @vhagor, @frankie-ys, @zufangzhu, @stmatengss, @seanwestfall, @kamb-code, @somuai, @Josephasafg, @Aj2280, @am-cohere, @Sip4818, @liuyao0322, @heliubj18, @krsish, @Oskii, @xijiaat, @benenzhu, @tdoublep, @akshaver, @SherifWaly, @TensorRaya, @shikamd123, @Roderick-Wu, @lcskrishna, @BANANASJIM, @soodoshll, @mlazos, @Akine-Ko, @miyakido, @meena-at-work, @nikhilkulkarni1755, @jcayab, @wincent8, @abrahamzewoudie, @ronensc, @haic0, @gc-fu, @ykamiset, @Xarbirus, @zvier, @avininjamay8, @ColinZ22, @YunzhuLu, @jiacao-amd, @kliuae, @tobymao, @zexplorerhj, @ywang96, @YoavMiron, @xonder, @bjf-frz, @cenab, @josiahdavis, @juliendenize, @JiataiWang, @colinmcnamara, @tlrmchlsmth, @LearningMachine621, @jamesETsmith, @YukioZzz, @kushaldabbe, @lzzzzzc, @SalikovAlex, @pmanczak, @fadara01, @JulienDarve, @nicole-lihui, @zRzRzRzRzRzRzR, @100milliongold, @labAxiaoming, @GuyStone, @Wauplin, @SorenDreano, @chengchengpei, @circlepen, @amd-xavierwang, @tahsintunan, @biswapanda, @junuxyz, @walterbm, @arcusbuilds, @JartX, @arpera, @jcotant-inferact, @PatrykSaffer, @maeehart, @surajm20061998, @Woolgathererer, @kebe7jun, @ShengleiFu, @pavelzak, @stu-cao, @CedricHwong, @lxy-alexander, @BWAAEEEK, @Chinmay-Kulkarni-AMD, @mgehre-amd, @BruceLoveDecimal, @hickeyma, @jiangkuaixue123, @xiaozhenbi, @ganeshr10, @Ronald1995, @arkohut, @russellb, @zqlcode, @tomasruizt, @thillai-c, @samnordmann, @Ianniu123, @Luosuu, @netanel-haber, @MicheleCampi, @zhaoguochun1995, @JohnQinAMD, @liranschour, @jacklin78911-collab, @migarci2, @404mario, @ItsRoy69, @mkhazraee, @Ubospica, @Thangnguyenvn98, @YeonwooSung, @alexliluz, @ECMGit, @luoyuctl, @averma12, @drslark, @ppalanga, @zwang86, @wangshangsam, @jiahanc, @vllm-agent, @connorcarpenter15
v0.29.0
Highlights
This release features 594 commits from 277 contributors (91 new)!
- Model Runner V2 is now the default for all models (#53183), completing the rollout that began with pooling models (#48290). MRV2 also gained CUDA graph memory profiling for KV cache auto-sizing (#53306), batch-sharded sampling that cuts per-step logits memory by 1/TP (#50465), prompt embeds (#42963),
extract_hidden_statesspeculation (#49811), padded FULL cudagraph dispatch for uniform decode under spec decode (#53407), and DP-sync skipping before EAGLE/MTP draft prefill (#53694). MRV1 remains in use for a few ROCm models and features MRV2 does not yet support. - New models: Hy4-preview, Tencent's 770B/49B-active MoE with Gated DeepSeek Sparse Attention and native MTP (#54160); Qwen3.8-Flash-Next with BF16/FP8/NVFP4 and MTP (#53896); GraniteSWA and GraniteMoeSWA (#52706); NemotronH_Omni_Reasoning_V3 with MTP (#52929, #53121); Kimi K3 NVFP4 checkpoints (#53132).
- Kimi-K3 and DeepSeek V4 performance: fused MXFP4 top-k finalization in the K3 latent tail (about 5% E2E latency, #53152), K3 Mamba metadata preparation in one Triton launch (6.6-7.6x kernel speedup, #52388), tuned Hopper low-latency GEMM (#54088) now also dispatched on SM100 (#53534) and used for
eh_proj(12.9-25.2% kernel speedup, #53942), GEMM-RS extended to GEMM-AR (#53053), MLA gate merged into the QKV-A projection (#54015), K3 DCP with DSpark (#52188) and DCP partial prefix cache hits (#50493); DeepSeek V4 shared experts fused into MegaMoE (#53040), adaptive top-k width re-landed (#52823), a native SwiGLU clamp kernel for Humming MoE (#53685), and an opt-in FlashInfermoe_epexpert backend (#49636). - Speculative decoding: per-request acceptance stats in OpenAI API responses via
--per-request-spec-decode-metrics(#48915), adaptive verification extended to logprobs (#52242), SM100 sparse MLA for GLM-5.2 (#52783) and DeepSeek V4 on SM90 (#52795), Qwen3-Omni DSpark drafts (#52560), PLaMo3 EAGLE-3/DFlash (#54239), and DFlash2 loading from speculators format (#53797). - RL weight sync: a new
sharded_rdtP2P backend where each worker pulls only its TP/EP slice over NIXL or Ray Direct Transport (#43375), rank-local IPC weight updates (#52497), sparse checkpoint-coordinate updates through native weight loaders (#50723, #53751), and routed expert loading for gpt-oss (#52209). - Mamba prefix caching: internal prefill checkpoints deliver a 9%-25% TTFT improvement (#52789);
prefix_cache_retention_intervalis now a CLI argument defaulting to 0 (#52216), with dense retention automatically restored for hybrid models using EAGLE/MTP (#55760, #55861). - New defaults: FlashInfer all-reduce enabled by default for TP CUDA groups, opt out with
VLLM_ALLREDUCE_USE_FLASHINFER=0(#52998); prefix-cacheNONE_HASHis deterministic by default so distributed KV cache users no longer need to pinPYTHONHASHSEED(#51875); new--max-num-queued-reqs/--max-num-queued-tokensadmission-control flags (#49445). - Breaking changes: ten deprecated model architectures removed (#53608); FlexOlmo, Olmo3 and Hunyuan V1/VL migrated to the Transformers modeling backend (#53615); PyAV video decoder backend removed (#54231);
python -m vllm.entrypoints.openai.api_serverdeprecated in favor ofvllm serve(#52131);VLLM_TEST_FORCE_FP8_MARLIN(#52182) andVLLM_ROCM_USE_AITER_FP4_ASM_GEMM(#53141) removed.
⚠️ Model Runner V1 Deprecation
Now that Model Runner V2 is used by default, we are considering Model Runner V1 deprecated and are targeting v0.32 for its removal. We do not intend to accept any more MRV1-specific improvements or optimizations.
Some features are not yet supported in MRV2 but we are planning for these gaps to be closed within the next 2-3 weeks. These include sequence parallelism, dual-batch overlap, elastic expert parallellism, custom logits processors and certain speculative decoding methods. For now, vLLM will still fall back to use MRV1 if any of these features are configured.
Release Artifacts
Python Wheels
| Platform | Install |
|---|---|
| PyPI (CUDA 13.0) | pip install vllm |
| PyPI (CUDA 13.0, uv) | uv pip install vllm --torch-backend=auto |
| ROCm | pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.29.0/rocm723 |
| XPU | uv pip install vllm --extra-index-url https://wheels.vllm.ai/0.29.0/xpu --extra-index-url https://download.pytorch.org/whl/xpu --index-strategy unsafe-best-match |
Docker Images
| Platform | Docker Image |
|---|---|
| CUDA 13.0 (Default) | docker pull vllm/vllm-openai:v0.29.0 (v0.29.0-cu130 also works) |
| CUDA 12.9 | docker pull vllm/vllm-openai:v0.29.0-cu129 |
| CUDA 13.0 + Ubuntu 24.04 | docker pull vllm/vllm-openai:v0.29.0-ubuntu2404 |
| CUDA 12.9 + Ubuntu 24.04 | docker pull vllm/vllm-openai:v0.29.0-cu129-ubuntu2404 |
| ROCm | docker pull vllm/vllm-openai-rocm:v0.29.0 |
| CPU | docker pull vllm/vllm-openai-cpu:v0.29.0 |
| XPU | docker pull vllm/vllm-openai-xpu:v0.29.0 |
Other Artifacts
Pre-built release artifacts are available in the Assets section at the bottom of this page, including:
- Source distribution tarball
- CUDA 12.9 Python wheels for x86_64 and arm64
- CUDA 13.0 Python wheels for x86_64 and arm64
- CPU Python wheels for x86_64, arm64, and macOS
- XPU Python wheel for x86_64
Model Support
- New models: Hy4-preview (#54160), Qwen3.8-Flash-Next (#53896), GraniteSWA and GraniteMoeSWA via the existing Granite implementation (#52706), NemotronH_Omni_Reasoning_V3 (#52929) with MTP for Nemotron VL models (#53121), bidirectional attention for DeepSeek-backbone embedding models (#52948), FP8 ModernBERT (#53101), and Kimi K3 NVFP4 checkpoints (#53132).
- Transformers modeling backend: FlexOlmo, Olmo3 and Hunyuan V1/VL migrated off native implementations (#53615), multimodal path hardened (#51827), and
RMSNormFuser.fuseperformance fixed (#52766). - Weight loading: weight tying now inspects the checkpoint so a real
lm_headis loaded even when the config claims tied embeddings (#51665, #53170). - LoRA: DeepSeek V4 (#53361), Qwen3-Omni multimodal LoRA (#52786, #53557), tower/connector LoRA for LLaVA-NeXT (#49788) and LFM2-VL (#51498), plus fixes for Muse-Glimmer (#53513), Qwen3.5 embedding modules (#48850), partial LoRA on Qwen3.5/3.6 GatedDeltaNet (#47640), int32 overflow in punica kernels at long context (#53034), false target matches on unsupported module types (#52313), and base-layer/routed-expert prefix ordering (#52552).
- Multimodal performance: ViT full CUDA graph for Idefics3 and SmolVLM (#47625), packed encoder attention for Pixtral (#52185), fused Kimi vision Q/K RoPE kernel (#50400), Dots3 NOTE runtime (#53517) and Omni encoder optimizations (#53460), MM tensors no longer broadcast to workers for prefix-cache-covered items (#52041), redundant placeholder scans skipped (#52925), common token sequences cached (#53560), async media resolved concurrently across modalities (#54537), and H2D copies pinned to avoid stream stalls (#53412, #54292, #54299).
- Multimodal inputs: video embeds accepted by the Python frontend (#54242), modality-scoped
mm_processor_kwargshonored (#53808), Qwen3-VL profiling honorscap_pixels_per_frame(#54380), video frame sampling respects MP4 edit-list trims (#48608), oversized items skip the MM processor cache instead of crashing (#53016), encoder cache entries survive until their last use (#54284), and repeated multimodal requests with the SHM cache no longer terminate the engine (#54994). - Correctness: PaliGemma stale image scaling removed (#52692), Mistral3 placeholder grid with processor size overrides (#52874), MiniCPM-o on Transformers v5 (#54501), JinaVL cache order (#53553), mixed CLIP/SigLIP pooling batches (#53165), XD-RoPE models on prefix-cache hits (#53456), Qwen3-Omni audio encoder with non-divisible TP (#50858), GLM-5.2 no longer uses dense MHA (#52512), Moondream3 MoE all-reduce (#54152), deepseek-vl2 config defaults (#51302), HY-V3 compressed-tensors ignore matching (#48682), MiniMax-M3 FP8 query allocation under CUDA graph replay (#51203), and GraniteMoeHybrid quantized expert loading (#54052).
Engine Core
- Model Runner V2: default for all models (#53183) and pooling models (#48290), CUDA graph memory reservation (#53306), batch-sharded sampling (#50465), prompt embeds (#42963),
extract_hidden_states(#49811), padded FULL cudagraph dispatch (#53407), DP-sync skipping for drafts (#53694), decoupled draft/target gumbel noise streams (#54282), encoder-only path split out (#53176), and memory released correctly on shutdown and sleep (#53508, #54246, #54162, #53955, #53682). - Speculative decoding: per-request acceptance stats (#48915), adaptive verification with logprobs (#52242), on SM100 sparse MLA (#52783) and DSv4 + SM90 (#52795), varlen trtllm-gen decode (#52157), FlashInfer MLA for DSpark drafting under DCP (#54277), fused GDN MTP for all Qwen head ratios (#52539), widest uniform decode batch captured by default (#50488) with memory-safe graph sizes (#54418), and fixes for short_conv/LFM2 targets (#50272), speculators-format config overrides (#42376), DFlash draft RoPE layout (#54373), draft models with a different GQA ratio (#53002), draft models with a larger hidden size under TP (#52193), DSpark backend inheritance scoped to DeepSeek V4 (#52809), generic
DSparkDraftModelconfigs for Qwen3 (#52197), and Gemma4 MTP under CUDA graphs (#53884). - Prefix caching & scheduling: Mamba internal prefill checkpoints (#52789),
prefix_cache_retention_intervalargument (#52216, #55760, #55861), deterministicNONE_HASH(#51875), queue admission control (#49445), KV null block reserved when validatingmax_model_len(#47272), spec decode no longer padded up tomax_model_len(#53962), and negative external block allocation prevented (#52707). - RL workflows:
sharded_rdtP2P weight sync (#43375), rank-local IPC updates (#52497), sparse checkpoint updates (#50723, #53751), gpt-oss routed expert loading (#52209), stable DeepSeek V4 mHC broadcast buffers across weight sync (#52626), and packed weight transfer stream reuse capping reserved-memory waste (#52951). - Determinism:
trace_decode_token_idsfor deterministic decode replay (#46701), per-arch tuned batch-invariant matmul configs (about 3x decode kernels on RTX 4090D/H20, #53247), Blackwell autotuning with 33.6% E2E latency reduction (#53649), deterministic MoE combine under DP+EP (#45683), andfuse_allreduce_rmsdisabled under batch invariance (#51292). - Kernels: fused embedding kernel (#53677), FA4 re-enabled for head_size=256 on Blackwell (#52980), vectorized sparse MLA mask loads (#52217), masked MHA prefill for GLM-5 head dimensions (#53785), GLM-5.2 sparse MLA Q concatenation fused with head padding (#53878), fused QK-norm + partial MRoPE + gate for Qwen3.6 (#52676), FlashInfer CuTeDSL BF16 low-latency GEMM as opt-in
--linear-backend flashinfer_cutedsl(#50572), replicated embedding and norm fusion for DSV3 flat models (#48484), standardized fused shared-expert selection (#51695), GPT-OSS MoE topk metadata reuse (#45457), tuned cooperative topk (#53382), and tuned FP8 fused_moe for Qwen3.5 on L40S (+7%, #53819). - Robustness: KV cache layout standardized under a
KVCacheLayoutenum (#51718), JIT warmup provider registry (#50174),--cpu-offload-paramsnow reaches vision/audio towers (#53120), attention backend probe failures no longer crash init (#51703), FlashInfer XQA falls back on unsupported head_dim (#53111), FlashInfer prefill LSE normalized before merging to fix prefix-cache logits divergence (#52796), seed preserved when a batch mixes seeded and unseeded requests (#51866), startup thread allocation accounts for local DP workers (#52385), int32 overflow fixes in fused SiLU block quant (#53409) and LoRA kernels (#53034), a shared-memory race in fused groupwise RMSNorm quantization (#54111), BLHNC addressing for FlashInfer sparse MLA (#54465), Mamba state copy race (#50729), and astart_profileno-op after auto-stop (#51839).
Hardware & Performance
- NVIDIA: DeepSeek V3.2 / GLM-5.2 DSA routed to the optimized CUDA path on all GPUs (#52861), PCP for DSv3.2 sparse MLA (#52046), cuBLAS out_dtype router GEMM on all archs including GB10 (#54048), FlashInfer all-reduce tuning on SM103 (#53318, #53606), FA4 hdim256 on SM100 (#52980), SM120 sparse MLA fixes (#51395, #53574), DeepSeek V3.2 fused kernel grids hardened for 65k+ token launches (#52381), FlashMLA sparse decode workspace fix (#53755), MNNVL Lamport corruption fix (#53000), and opt-in Rubin Docker builds for CUDA 13.4/13.5 (#53443).
- AMD ROCm: dual-stream decode with hipgraphs (#52033), W4A4 preshuffled asm GEMM by default (+15% throughput on Llama-3.3-70B MXFP4, #53141), ROCr/CLR update fixing graph replay segfaults with up to about 20% TPOT improvement (#53712), fused KDA decode on MI325X (#52293), FULL cudagraphs for AITER MLA spec decode (#51171), FP8 asm MLA prefill for non-divisor head counts (#51040), DCP causal multi-token verification (#51705) and prefix cache hits for Kimi-K3 (#53598), AITER PA gluon decode for MiniMax-M3 (#52849), DeepSeek-V4 fusions for mHC/RMSNorm (#52737), C4A top-k (#52882), C4 compressor GEMMs (#53838) and SWA q/kv norm + FP8 quant (#53540), fused shared experts for block-FP8 (#53097), CPU offload on ROCm 7.13+ (#43018), TheRock 7.14 preview docker (#49925), int4/int8 quantization fixes (#52112, #48998, #51632, #53110), CUDA graphs captured on the current stream (#53818), AITER metadata preserved across graph replay (#53821), and improved ROCm detection under WSL (#38434).
- Intel XPU: INC int4 W4A8 linear backend (#50501), AutoRound MXFP8 MoE (#51248), EC connector KV offloading (#49532), and fixes for HunyuanOCR XD-RoPE (#52174), sparse-MLA metadata sync (#52066), Mamba state pointer overflow (#48109), oneCCL warm-up at world size 1 (#52389), and MRoPE (#53201).
- CPU: AMX high-performance MLA backend for DeepSeek V2/V3/R1 (#52616) with MLA now running end-to-end (#51471), FP16/BF16 persisted GDN state on AMX (#52191), Int8 MoE through zentorch on AMD Zen (#44834), Voxtral support (#53921), C++ causal_conv1d GDN on non-AMX AVX-512BF16 (#49688), and assorted fixes (#54042).
Large Scale Serving
- Context parallelism: Kimi-K3 DCP with DSpark (#52188) and partial prefix cache hits (#50493), FlashInfer native CP for MLA decode (#54012), FlashMLA sparse DCP on Hopper with MTP (#46514), NIXL P/D DCP for MLA models (#50611), PCP for DSv3.2 (#52046) and NIXL PCP producers (#52779),
--dcp-q-replicatewith query replication default-on for GLM sparse attention (#50382), DCP fused attention fix for DeepSeek-V3.2 / GLM-5.2 (#50005), sparse MLA metadata and kernel block sizes under DCP (#52377, #51031), PCP PIECEWISE cudagraph fixes (#53869, #53515), and PCP compatibility checks delegated to the PCP manager so plugins can enable GQA+PCP (#53853). - Elastic EP: reduced eager-mode reconfiguration downtime (#51885), AOT cache reuse preserved during scaling (#53378), and scale-below-minimum rejected (#52702).
- MoE communication: DeepEP v2 receiver CPU overhead (#51114) and MXFP8 activation scale dispatch (#51398), FlashInfer one-sided All2All refinements (#51924), DeepEP v2 fixes for
--enforce-eagerstartup (#51824) and the decode/cudagraph path (#52632), NCCL>=2.31 heap overflow fix (#53008), cross-node MNNVL all-reduce gated by capability (#53253), and MNNVL all-reduce buffers sized for DSpark (#50932). - KV connectors: Mooncake Store decode KV saving via
save_decode_cache(#52466) and hybrid DCP prefix caching (#53324), externally transferable KV cache group identification (#53779), async KV loads deferred past forward launch (#53333),kv_transfer_paramsfor/inference/v1/generate(#42644), MoRIIO shared KV region registration after the layout refactor (#53698), and Mamba fixes for Mooncake (#51362, #51358, #53663) and NIXL (#53523). - KV offloading: EC offloading connector driven by CUDA events (#49994), ownership in KV cache events (#52067, #52068), P2P tier request-level offload (#52912) and abort handling (#52571),
/dev/shmleak on crash fixed (#52596), CPU->GPU loads ordered against compute stream (#50696),store_thresholdcounting fixed (#52227), in-flight primary keys cascaded (#53329), and padded GPU cache storage handled (#54021). - Data/pipeline parallel: PP silent corruption fix (#54962, #49274), DP coordinator wake handling (#51481), device sync on pause (#52914), TCPStore port fixes for Ray (#53666, #50969), DP supervisor inheriting the uvicorn config (#52473), EPD encoder round-robin fix (#52491), and producer-only EC config normalization (#53656).
Quantization
- New backends: FlashInfer TRT-LLM MXFP8 linear (#52204), b12x FP4 MoE for SM120/SM121 (#52018), AutoRound block-wise FP8 (#47434), Humming MoE with MXFP4 weights + block-FP8 activations (#51332), and Humming for compressed-tensors WNA16 MoE (#48918).
- Fixes: weight-only NVFP4 checkpoints routed through W4A16 (#54427), compressed-tensors block FP8 with Marlin (#52966) and int8 grouped WNA16 MoE (#52002), OCP MX mxfp6 activation emulation (#52704), MXFP8 FlashInfer path guarded on availability (#52648), and Humming activation aliasing (#54056).
API & Frontend
- New endpoints & options:
/v1/messages/renderfor the Anthropic Messages API (#45803),/cohere/v2/chat/render(#53219), SSE keep-alive comments for idle streams (#51034), per-request spec decode metrics (#48915), video embeds input (#54242),--max-num-queued-reqs/--max-num-queued-tokens(#49445), and pooling requests can set padding (#51157). - Anthropic & Cohere:
vllm_xargsforwarded to sampling params (#53308),stop_sequencestop reason reported (#45807), Cohere citation/tool 500s fixed (#52175), and stop string limit applied (#53750). - OpenAI compatibility:
logprobs=-1in Completions (#46175), streamed logprob offsets with echo (#47815), batched chat echo (#52529), all choices returned from/inference/v1/generatewith n>1 (#52399),cache_saltforwarded for content parts (#54315),stop_token_idsvalidated against vocab (#54196), and 4xx for client errors in/detokenize(#52622), malformed namespace tools (#53763), malformed base64 audio (#53744), and unknown chat roles in DeepSeek encoders (#53071); tool-call arguments in replayed history are parsed defensively (#48922), emptyFlatLogprobsslices are handled (#53704), and empty bad-word tokenizations are rejected (#53433). - Structured output & parsers: reasoning-end detection scoped to the current turn (#54089), terminal grammars stop under
min_tokens(#54218), unsupported pattern+length schemas rejected (#49996), MistralCommonBackend tokenizers (#52720), XGrammar termination in batches (#52805), spurious FSM errors after speculative reasoning end (#53046), shared parser engine adapters (#52830), Gemma4 parenthesized tool calls (#53657) andenable_thinkingdefault (#52430), HY-V3 parallel calls in one delta (#53965), GPT-OSS Harmony strict grammar (#52222), Kimi K3 reserved markers excluded from response text (#52889), and unused request-local reasoners skipped (#52573). - Rust frontend: HY3 unified parser with local XGrammar structural tags (#53054), gRPC audio/video inputs (#53760), gRPC LoRA lifecycle control (#52840, #52031, #53756),
--generation-config vllm(#53044),truncate_prompt_tokens(#48584), OpenAI edge-case alignment (#53218), optimized SSE hot path (#51321), pure-Rustprotoxreplacingprotoc(#52892), RL world-size reporting (#53204) and routed expert prompt offsets (#52703), and fixes for GLM-5.2 template parity (#51426), Qwen parser auto-detection (#51169), Kimi K3reasoning_effort="none"(#53043),n > 1rejection on/inference/v1/generate(#52844), and theLogprobsTensorswire schema (#53939). - Pooling: BGE-M3 throughput (+3.13%, #53464) and task validation (#51823), prompts truncated before padding (#54364), parallel arrays truncated with the prompt (#54407, #54509), and batched input throughput in
vllm bench serve(#53213). - CLI & tooling:
vllm launchruns the serve argument checks (#52825),run_batch.pymoved out of the openai folder (#53500) with its upload retry fixed (#50588),vllm benchwarns on a warm prefix cache for random runs (#53920) and restores multimodal datasets on thevllmthroughput backend (#52168), and the vLLM recipes tool gained sweep recommendations and alias parsing (#53325, #53946).
Security
cache_saltlength bounded to 1024 to prevent scheduler CPU exhaustion (#54353).- Oversized media rejected before full download (#51896);
VLLM_MAX_AUDIO_CLIP_FILESIZE_MBenforced on all audio paths (#53561). - Decoder prompt-length validation enforced for processors that skip the check (#46588); PyNvVideoCodec decoder slot limit bypass fixed (#52126).
api_keyandhf_tokenredacted from startup logs, compile cache factors, and the Rust frontend launch log (#52523, #53625, #53738).
Dependencies
- FlashInfer 0.6.18 (#54313), huggingface-hub 1.28.0 (#52797), tpu-inference v0.28.0 (#54020), NIXL 1.3.2 (#51777).
- InstantTensor added to CUDA dependencies; the loader is not enabled by default (#52801). CuPy constraint relaxed to exclude only 14.1.0 (#44284).
- ROCm base image: ROCr/CLR update (#53712), rocprofiler-sdk 1.3.2 (#53182), TheRock 7.14 preview (#49925), LMCache connector packages (#51208).
- XPU: vllm_xpu_kernels 0.1.14.1 (#54203), UCX install updated (#53817).
- Build fails closed when the selected precompiled CUDA variant is unavailable (#52545).
Breaking Changes & Deprecations
- Ten deprecated architectures removed: Arctic, Chameleon, Cheers, Fairseq2Llama, FireRedLID, GritLM, HCXVision, MPT, the
RWForCausalLMandStableLMEpochForCausalLMaliases, andPrithviGeoSpatialMAE(superseded byTerratorch) (#53608). - FlexOlmo, Olmo3, Hunyuan V1 and Hunyuan VL are now served through the Transformers modeling backend (#53615).
- PyAV video decoder backend removed; use OpenCV or Torchcodec (#54231).
python -m vllm.entrypoints.openai.api_serveris deprecated; usevllm serve(#52131).- Model Runner V2 is the default runner for all models (#53183).
prefix_cache_retention_intervaldefault changed from dense to 0 for SWA/SSM models; the env var is deprecated in favor of the argument (#52216).- FlashInfer all-reduce enabled by default (#52998).
VLLM_TEST_FORCE_FP8_MARLINremoved in favor of--linear-backend/--moe-backend(#52182);VLLM_ROCM_USE_AITER_FP4_ASM_GEMMremoved (#53141); dead--attention-config.use_prefill_decode_attentionremoved (#52557); other long-deprecated parameters cleaned up (#53559).
New Contributors
- @030611 made their first contribution in #51823
- @92hyungjun made their first contribution in #47272
- @ActiveSky made their first contribution in #52692
- @adisivaprasad made their first contribution in #53854
- @Agoni-02 made their first contribution in #48850
- @AmitMY made their first contribution in #48608
- @Andy365-365 made their first contribution in #52523
- @andyluo7 made their first contribution in #53821
- @AnkitNakhawa made their first contribution in #52491
- @anmolgupt made their first contribution in #52874
- @bobboli made their first contribution in #51924
- @brianosaurus made their first contribution in #52557
- @CalvinXKY made their first contribution in #50858
- @canlahlah made their first contribution in #53409
- @CherryLemon made their first contribution in #53877
- @CHIPMUNK-T0T made their first contribution in #47625
- @cogniera made their first contribution in #53839
- @cr-zhao made their first contribution in #52385
- @daviswer made their first contribution in #52706
- @DCoEngine made their first contribution in #48682
- @dineshchitlangia made their first contribution in #49688
- @dkrisman made their first contribution in #54380
- @dmvevents made their first contribution in #52632
- @Edge-Explorer made their first contribution in #53435
- @eilamc14 made their first contribution in #47640
- @eligotts made their first contribution in #54315
- @Eoin-Houstoun made their first contribution in #51703
- @floatlibai made their first contribution in #52144
- @fuzzifikation made their first contribution in #51034
- @hagaikwa-redhat made their first contribution in #53230
- @haoyangqian made their first contribution in #52690
- @Hert4 made their first contribution in #51157
- @ima-helikoptaaa made their first contribution in #54427
- @jbyczkow made their first contribution in #52174
- @JC-ut0 made their first contribution in #53071
- @jiahaoliang made their first contribution in #51262
- @JiataiWang made their first contribution in #53553
- @jl9876 made their first contribution in #51248
- @JulianZJN made their first contribution in #53253
- @jungjiyu made their first contribution in #53939
- @kyleliang-nv made their first contribution in #51203
- @LH-and-FPGA made their first contribution in #52648
- @li-ukumar made their first contribution in #52571
- @LioEinaudi made their first contribution in #53247
- @LironKesem made their first contribution in #53921
- @Lossfull made their first contribution in #52948
- @lxy-alexander made their first contribution in #52430
- @lxyxinyi made their first contribution in #50809
- @machero made their first contribution in #53531
- @matthewkotila made their first contribution in #48915
- @mhoqueanik made their first contribution in #49636
- @mhuzaifa3 made their first contribution in #53625
- @minjang made their first contribution in #53530
- @MKQuantum made their first contribution in #51866
- @new-TonyWang made their first contribution in #53704
- @nicholaskh-ai made their first contribution in #53819
- @oliverholworthy made their first contribution in #52185
- @prakharPant made their first contribution in #52743
- @Prudhvivuda made their first contribution in #53016
- @rajathpi made their first contribution in #52622
- @ray24777 made their first contribution in #53120
- @roachsinai made their first contribution in #53528
- @RookieCoder-Camera made their first contribution in #49274
- @sandeep-maddipatla made their first contribution in #51777
- @seonjinn made their first contribution in #52204
- @ShengleiFu made their first contribution in #53650
- @shepark made their first contribution in #51302
- @shijuzhao made their first contribution in #45683
- @shipiyouniao made their first contribution in #51979
- @ShuaiShao93 made their first contribution in #53034
- @simon-veitner-redhat made their first contribution in #52980
- @sseanliu made their first contribution in #52041
- @studioego made their first contribution in #52389
- @tanchao made their first contribution in #50191
- @thanhpt1110 made their first contribution in #52720
- @theamalsebastian made their first contribution in #52588
- @thunguo made their first contribution in #53101
- @tolleybot made their first contribution in #51292
- @tommy-asai-sonarsource made their first contribution in #51395
- @tthakkal made their first contribution in #50501
- @ukannika made their first contribution in #52849
- @VBS2004 made their first contribution in #48922
- @wyettzeng made their first contribution in #52209
- @Xuan-1998 made their first contribution in #53008
- @y0hnn made their first contribution in #52002
- @Yiqin-17 made their first contribution in #52925
- @YukioZzz made their first contribution in #51705
- @ZHIHANCHEN03 made their first contribution in #53432
- @Zhou248 made their first contribution in #52560
- @zllion made their first contribution in #46701
- @zwischenraum made their first contribution in #50272
Contributors
@AndreasKaratzas, @khluu, @mgoin, @taneem-ibrahim, @yewentao256, @njhill, @BugenZhao, @hmellor, @GirasoleY, @DarkLight1337, @mayuyuace, @WoosukKwon, @jeejeelee, @noooop, @jperezdealgaba, @LucasWilkinson, @wzhao18, @stefankoncarevic, @gau-nernst, @okorzh-amd, @ZJY0516, @HollowMan6, @connorcarpenter15, @gcanlin, @aoshen02, @linitra24, @mhuzaifa3, @Isotr0py, @Etelis, @MatthewBonanni, @TheEpicDolphin, @zupengwang, @atalman, @NickLucche, @rasmith, @elvircrn, @JasonKeyiL, @Hotragn, @waizuichougou, @pmanczak, @divakar-amd, @he-yufeng, @xuebwang-amd, @qgallouedec, @esmeetu, @ZeldaHuang, @Prudhvivuda, @chaunceyjiang, @zxd1997066, @louie-tsai, @andyxning, @pisceskkk, @tjtanaa, @qli88, @tlrmchlsmth, @LopezCastroRoberto, @micah-wil, @hongxiayang, @BabyDrangoner, @yimdev, @mganczarenko, @SageMoore, @biswapanda, @sfeng33, @russellb, @itayalroy, @bigPYJ1151, @drakosha, @qwerqwerqwe8688-jpg, @akii96, @shen-shanshan, @meiyeh123, @reidliu41, @yma11, @frgossen, @lukealonso, @tanchao, @Fangzhou-Ai, @vllm-agent, @djramic, @andrewbcohere, @zyongye, @gty111, @ShuoleiWang, @ZHIHANCHEN03, @KurodaKanbei, @alexeldeib, @khushali9, @xiaohuguo2023, @hungnnvidia, @thisjiang, @arpera, @ShengleiFu, @zhenwei-intel, @jungjiyu, @yzong-rh, @charlifu, @hclsys, @Ronald1995, @mkhazraee, @YukioZzz, @theamalsebastian, @030611, @cr-zhao, @jbyczkow, @tommy-asai-sonarsource, @jikunshang, @bobboli, @LH-and-FPGA, @SayHelloToWorld, @kkt-cohere, @floatlibai, @rajathpi, @lxy-alexander, @gangula-karthik, @ActiveSky, @fxmarty-amd, @sstamenk, @mpashkovskii, @LiuYinfeng01, @chaojun-zhang, @Oxygen56, @daviswer, @vineethsaivs, @Agoni-02, @Andy365-365, @wangxiyuan, @haoyangqian, @Naveassaf, @eilamc14, @sseanliu, @y0hnn, @dineshchitlangia, @yiliu30, @Lossfull, @lxyxinyi, @vanshbhatia-amd, @jcotant-inferact, @chengy-sysu, @92hyungjun, @xyang16, @AmitMY, @zllion, @AnkitNakhawa, @dmvevents, @lengrongfu, @JC-ut0, @Eoin-Houstoun, @sagearc, @wjabbour, @sandeep-maddipatla, @seonjinn, @Yiqin-17, @kyleliang-nv, @thanhpt1110, @MKQuantum, @matthewkotila, @vhagor, @tthakkal, @ShuaiShao93, @hagaikwa-redhat, @shijuzhao, @elwhyjay, @Gregory-Pereira, @ErenAta16, @morrison-turnansky, @DCoEngine, @SoluMilken, @xianbaoqian, @kliuae, @nascheme, @waynehacking8, @stecasta, @vMaroon, @zwischenraum, @Rohan138, @Zhou248, @brianosaurus, @anmolgupt, @wyettzeng, @hao-aaron, @frank-suwen, @danisereb, @fuzzifikation, @KernelClint, @thunguo, @studioego, @shipiyouniao, @cjackal, @LioEinaudi, @guan404ming, @libinta, @mhoqueanik, @minjang, @Edge-Explorer, @shepark, @JiataiWang, @jiahaoliang, @therealnaveenkamal, @ColinZ22, @tolleybot, @almogtavor, @simon-veitner-redhat, @hyeongyun0916, @nicholaskh-ai, @mawong-amd, @adisivaprasad, @ray24777, @robertgshaw2-redhat, @new-TonyWang, @oliverholworthy, @fanxingran, @lucianommartins, @hallerite, @avininjamay8, @VBS2004, @omerpaz95, @Sunt-ing, @Hert4, @wangshangsam, @cogniera, @jeffreywang88, @Xarbirus, @fynnsu, @jiahanc, @zixi-qi, @prakharPant, @haic0, @maobaolong, @xinyu-intel, @CHIPMUNK-T0T, @hangy-amd, @afriedri, @canlahlah, @Alnusjaponica, @ukannika, @pranavthakur0-0, @zzaebok, @Xuan-1998, @JulianZJN, @roachsinai, @ivanium, @CalvinXKY, @yu-xin-c, @rchalamala, @QwertyJack, @Kaif10, @yudigege86, @yiz-liu, @dkrisman, @simondanielsson, @machero, @CherryLemon, @ima-helikoptaaa, @RookieCoder-Camera, @LironKesem, @peakcrosser7, @eligotts, @foraxe, @zufangzhu, @tianmu-li, @jl9876, @lucifer1004, @liranschour, @maithilijoshi20, @li-ukumar, @andyluo7, @Zhenzhong1, @faaany, @joerowell, @juhi10071998, @ganeshr10, @Priyjain-amd, @SubSir, @luyixiao95, @djw8605, @codex
v0.28.0
Highlights
This release features 584 commits from 270 contributors (76 new)!
- Kimi-K3 performance push: a major optimization effort for Kimi-K3 across the stack — Decode Context Parallel (DCP) support (#50484), fused FlashKDA decode and prefill kernels (#50654, #51311, #52458), SiTU activation support for MegaMoE (#50510), GEMM-RS for sequence parallelism (#52079), combined all-gathers with 1.5~3x kernel-level speedup (#51070), an adaptive speculative token budget delivering ~60% better DSpark TTFT (#51725), and optional shared-expert sharding saving ~17 GiB of memory per GPU (#50912). Kimi-K3 also now runs on ROCm with the V2 model runner (#51653).
- DeepSeek V4: sparse MLA now works end-to-end for plain decode, MTP, and DSpark speculative decoding (#51538), joined by AMD Quark NVFP4 support (#47972), reasoning-effort prompts and mappings (#50580), sparse top-k metadata kernel optimizations (#52084, #51967), narrowed eager CUDA graph regions (#51430, #52401), and ROCm enablement on gfx11 and gfx950 (#47017, #52212).
- Speculative decoding advances: DFlash2 with local convolution and a candidate selector (#52816), DSpark confidence-scheduled verification (#47808), and async scheduling auto-enabled for draft models (#48341).
- Model Runner V2 maturation: E/P/D disaggregation (#38390), weight offloading (#51413), multi-layer MTP KV cache support (#50062), encoder CUDA graphs (#49852), decoder token-wise pooling (#50931) plus Transformers pooling models (#52425), attention-free models (#52374), and
thinking_token_budgetsupport (#46727). - Tiered KV cache offloading: disk offloading support (#49644), out-of-tree secondary tier managers via
module_path(#51007), partial secondary-tier load results (#50321), tiering metrics (#48798), and a canonical CPU layout for parallelism-agnostic offload (#48414). - Rust frontend & gRPC: a standalone renderer (#50289), multimodal image inference over gRPC (#50368), explicit data-parallel rank routing (#51178), and RL lifecycle control (#51316), with protobuf schemas now published to Buf (#51276).
- New defaults:
max_num_batched_tokensraised from 8192 to 16384 (#51726), prefix caching enabled by default for Mamba models (#50991), and the Blackwell CUDA graph capture default raised to 1024 (#49390). - Breaking changes: bitsandbytes support migrated to an out-of-tree plugin (#43529); Transformers bumped to 5.15.0 (#51668); the deprecated
calculate_kv_scalesruntime KV scale calculation was removed (#49389);override_attention_dtypewas removed (#48684).
Release Artifacts
Python Wheels
| Platform | Install |
|---|---|
| PyPI (CUDA 13.0) | pip install vllm |
| PyPI (CUDA 13.0, uv) | uv pip install vllm --torch-backend=auto |
| ROCm | pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.28.0/rocm722 |
Docker Images
| Platform | Docker Image |
|---|---|
| CUDA 13.0 (Default) | docker pull vllm/vllm-openai:v0.28.0 (v0.28.0-cu130 also works) |
| CUDA 12.9 | docker pull vllm/vllm-openai:v0.28.0-cu129 |
| CUDA 13.0 + Ubuntu 24.04 | docker pull vllm/vllm-openai:v0.28.0-ubuntu2404 |
| CUDA 12.9 + Ubuntu 24.04 | docker pull vllm/vllm-openai:v0.28.0-cu129-ubuntu2404 |
| ROCm | docker pull vllm/vllm-openai-rocm:v0.28.0 |
| CPU | docker pull vllm/vllm-openai-cpu:v0.28.0 |
| XPU | docker pull vllm/vllm-openai-xpu:v0.28.0 |
Other Artifacts
Pre-built release artifacts are available in the Assets section at the bottom of this page, including:
- Source distribution tarball
- CUDA 12.9 Python wheels for x86_64 and arm64
- CUDA 13.0 Python wheels for x86_64 and arm64
- CPU Python wheels for x86_64, arm64, and macOS
Model Support
- New models: Muse Glimmer (#51655), Ling 3.0 Flash with BF16, MTP, and parser support (#51045) plus an FP8 variant (#51265) and hybrid MXFP4 routed experts (#52114), Dots3 NOTE native multimodal support (#51255), and Interns2mobius (#51149).
- Qwen: Qwen3.8 enabled on AMD ROCm (#50068), fused CUDA post-conv MTP decode kernel for Qwen3.5 GDN (#51674), GDN gates aligned with speculative tokens (#51812), and Qwen3.5 fixes for text-only checkpoints (#50734, #50355).
- Transformers modeling backend: MLA support (#48250), hardware-agnostic model definition (#49458), fully generalized input embedding handling (#51247), logit softcapping (#52173), and a hardened multimodal path (#51408, #51657).
- LoRA: vision tower LoRA for Gemma4 (#42662), tower/connector LoRA for Keye (#51780) and Ultravox (#48215).
- Vision encoders: ViT full CUDA graph for Kimi-K2.5 (#50929) and Ernie-4.5-VL (#45254, #51461), torch.compile for the Qwen3-VL encoder (#40116), and long-blocking H2D copies avoided in ViT (#51841).
- MoE: extended EPLB support for Mistral Large 3 and additional MoE backends (#48355), CuTe DSL skinny GEMM extended to GLM-5.2 (#49791).
- Speculative decoding coverage: EAGLE3 support declared on KimiLinear (#52171), Qwen3.6 dSpark acceptance coverage (#51310).
- Correctness: MiniMax-M3 NVFP4 inference (#48929) and compressed-tensors FP8 MoE SwiGLU params (#46845), Gemma3n/Gemma4 variable-length audio batch padding (#50958), Gemma 4 compatibility with the upcoming Transformers version (#49797), and a Qwen3-Omni crash on video without an audio track (#48420).
- Multimodal performance: fused on-device multimodal preprocess normalization (#50411), faster placeholder and token-match scanning (#50716), and repeated prompt-update scans avoided (#51774).
Engine Core
- Speculative decoding: DSpark confidence-scheduled verification (#47808), top-k DSpark Markov projection (#49969), DFlash2 with local convolution and a candidate selector (#52816), async scheduling auto-enabled for draft models (#48341), fused MTP trailing all-reduce with local-argmax draft tokens (#49793), and an adaptive budget for speculative scheduled input tokens (#51725).
- KV cache & scheduling: per-request scheduling for MLA chunked context (#50613), partial-tail prefix reuse with fine-grained prefix matching (#50507), backend-published KV packing in the KV-cache layout refactor (#51612, #51704), LIFO free-block reuse order restored when prefix caching is off (#51482), and silent request skipping in priority scheduling fixed (#49206).
- Performance: continued elimination of GPU<->CPU syncs on the execution path (#51458, #51738, #52369) now guarded by a CI sync check (#43107), new JIT warmup infrastructure with predicate filtering (#49315), the top-k/top-p Triton sampler launched with 8 warps (#51507), detokenization skipped in offline beam search (#50333), Mask Replay (#49577), optimized long-context MLA cache gathers (#51739), and HF revisions resolved to a commit hash once per model load (#49990).
- Hybrid/Mamba: prefix caching on by default (#50991), the final part of the Mamba attention module refactor (#44857), 3D-grid tiling of the state-copy Triton kernels (#49436), and Mamba alignment applied before encoder caps (#51603).
- RL workflows: stateful trainer send over NCCL and sparse NCCL (#50902),
CuMemAllocator.discard()for tag-selective GPU memory release (#52514), level-2 sleep/wake/reload fixed with LoRA enabled (#39935), and rewritten weight-transfer docs with standardized examples (#51729). - Startup robustness: file:// rendezvous for single-node executors eliminates startup port races (#50999, #51652), frontend processes are watched during engine startup (#43417), a
get_open_port()livelock on DP-reserved ports was fixed (#50965), and NVML is no longer re-initialized on every device-capability check (#50393).
Hardware & Performance
- NVIDIA: FlashInfer XQA decode support on SM12x (#49718), a CuTeDSL fused query kernel on SM100 (#49792), programmatic dependent launch for the DSA decode kernels (#50230), the native DSA decode path for MTP=3 on SM90 (#52164), GB10 fused-MoE FP8 tuning configs (#52502), and B12X dense linear backends (#52016).
- AMD ROCm: torch 2.12 / triton 3.7 stack bump (#50607), AITER and FP8 inference enabled on GFX120x (#43615), DeepSeek-V4 on gfx11 (#47017), optimized Triton sparse-MLA decode on gfx950 (#52212), FlyDSL decode-attention kernel for 4-bit TurboQuant KV cache (#47896) and an fp8 MQA logits kernel on gfx942 (#49544), a fused Kimi-K3 KDA decode kernel (#50654), fused bf16→fp32 router GEMM (#50268), pinned memory on supported WSL2 kernels (#50126), and preshuffled sparse indexing for 16-token blocks (#51216).
- Intel XPU: a torch linear backend including blockwise GEMM (#49664, #50826), MXFP8 linear weights for the INC DeepSeek V4 model (#48476), async-scheduling PP sampled-token broadcast overlapped with compute (#51650), an XPU wheel added to the release pipeline (#52108), tuned Mamba SSU configs for Arc Pro B70 (#50534), and UVA weight offloading fixes (#51770).
- CPU: an MLA backend so DeepSeek-V2/V3 can run on CPU (#49453), a triton-cpu wheel (#52092), GPTQ and AWQ enabled on s390x (#51148) along with tcmalloc (#50841), BF16 MoE routed through zentorch on AMD (#44201), an unquantized MoE backend for Power (VSX) (#51624), unquantized MoE migrated to the modular-kernel experts structure (#50133), and the MXFP4 block scale folded in 2 instructions instead of 4 (#51583).
Large Scale Serving
- E/P/D disaggregation: Model Runner V2 E/P/D support (#38390), duplicate image preprocessing removed with GPU-side preprocessing (#50390), KV consumers may omit multimodal embeddings (#52697), encoder-instance requests kept alive until their images are encoded (#50275), and EC connector scheduler/worker metadata plumbing (#49579, #49585).
- KV offloading: disk offloading for SimpleCPUOffloadConnector (#49644), out-of-tree secondary tier managers via
module_path(#51007), partial secondary-tier load results (#50321), tiering offloading metrics (#48798), data-parallel topology exposed to offloading backends (#51879), a canonical CPU layout for parallelism-agnostic offload (#48414), and quadratic ARC batch eviction avoided (#50992). - Mooncake: store group semantics (#44956), tenant ID support (#48069), and official wheels in the Docker image (#51067).
- Connectors: transfer mode (push/pull) included in the NIXL compatibility hash (#50620), a MoRIIO per-layer READ-completion barrier (#48534), 2P2D wide-EP with mori-ep/mori-io at dp=ep=16 (#45043), and stale remote cleanup in the push connector (#50234).
- Parallelism: EPLB balancedness calculation fixed and tested (#51813), dense multinode DP rescope (#49212).
Quantization
- Online quantization: online MXFP4 support (#49347), online weight scales shared across TP (#49764), precision preserved in online NVFP4 expert packing (#50029), and the online NVFP4 MoE kernel reused across reloads (#50074).
- NVFP4: batch-invariant NVFP4 MoE via CUTLASS (#40372), KV 4-over-6 scale search (#45187), CuTeDSL MoE with SwiGLU-OAI and ReLU2 activations (#47106), and out_dtype matched to the model dtype (#48861).
- New kernels: block-wise scaled_mm (#49932), DeepSeek-V4 AMD Quark NVFP4 with an emulation kernel (#47972).
- Fixes: dynamic INT8 W8A8 MoE config no longer built as W8A16 (#50833) and a TritonExperts crash (#51411), MXFP4 conversion for FlashInfer CUTLASS (#51038), fp32 weight scales and per-expert checkpoint mapping for MXFP4 (#51419), and fused block-scale orientation (#50727).
API & Frontend
- New capabilities: request priority parsed from an HTTP header (#51089), session ID plumbing into requests (#48048),
count_reasoning_tokensin the streaming parser engine (#45802),content_partson/inference/v1/generate(#51478),modeloptional on all/derenderrequest classes (#51463), output token IDs logged at DEBUG level (#52098), and vLLM Recipes connected to native config-based deployment and benchmarking (#51308, #51878). - Rust frontend: a standalone renderer (#50289), gRPC multimodal image inference (#50368), explicit data-parallel rank routing (#51178), RL lifecycle control (#51316), dynamic tools from developer messages (#51144), protobuf schemas published to Buf (#51276), and MiniJinja upgraded to 2.22 (#51235).
- Anthropic API: 4xx returned for client-caused errors on
/v1/messages(#52246),disable_parallel_tool_usepreserved (#52021), and stop sequences bounded (#51997). - Cohere: upstreamed parser fixes (#51998), stop sequences reported correctly (#51556), and vectorized binary embedding bit-packing (#52277).
- Structured output: request stop tokens masked in xgrammar until the grammar terminates (#49227, #50595), NUL bytes rejected in
structured_outputs.regex(#51796), negative token IDs rejected as out-of-vocabulary (#51795), andVLLMValidationErrorraised from validators (#52394). - Robustness: uvicorn signal handlers disabled instead of racing them (#50916), a consolidated entrypoint exception handler (#52261), 400 instead of 500 on non-object JSON bodies (#51654, #52528), generation inputs bounded before expensive work (#51447), and
cache_saltnow required to be non-empty (#50816).
Security
- Fixed a DoS via sample-rate forgery that bypassed the audio decode duration guard (#49948); the audio decode duration limit is now also enforced in NanoNemotronVL (#50221).
- DeepStream classified as a GPU backend with pixel limits enforced (#50755).
_load_ov2_processorguarded withresolve_trust_remote_code(#52952).- Documentation now warns that
--api-keydoes not gate all endpoints (#51999).
Dependencies
- Transformers 5.15.0 (#51668), huggingface-hub 1.27.0 (#51422), fastsafetensors upgrade (#50827).
- ROCm: torch 2.12, triton 3.7, torchaudio, torchvision (#50607).
- Runtime image upgraded to Ubuntu 24.04, picking up rdma-core > 44 (#51058).
- DeepGEMM pinned to the deepseek-ai nv_dev tip (#52035), DeepEP pinned by full commit hash (#52028), FlashAttention 3 built with the torch stable API (#49599).
Breaking Changes & Deprecations
- bitsandbytes support is now an out-of-tree plugin (#43529).
- The deprecated
calculate_kv_scalesruntime KV scale calculation was removed (#49389). override_attention_dtypewas removed (#48684).reasoning_contentoutput removal is documented as a breaking client change (#50624).- KV offload tiering metrics renamed from
kv_offload_tiering_block_{queries,hits}to..._chunk_...(#52812). - MoE legacy code removed (#51078).
New Contributors
- @acheamponge made their first contribution in #49353
- @acmore made their first contribution in #51259
- @anhtra3889 made their first contribution in #51002
- @anujbolewar made their first contribution in #50867
- @arthurgao2003 made their first contribution in #48215
- @baodii made their first contribution in #51215
- @bitborne made their first contribution in #44956
- @bohnstingl made their first contribution in #49458
- @ccaadaro made their first contribution in #51583
- @cmiyai made their first contribution in #46870
- @d4l3k made their first contribution in #51097
- @dmai-afk made their first contribution in #51120
- @dmholtz made their first contribution in #50977
- @efschu made their first contribution in #50734
- @fangchenli made their first contribution in #52277
- @fanxingran made their first contribution in #51011
- @fatday made their first contribution in #48171
- @fattchris made their first contribution in #48861
- @fcui-amd made their first contribution in #50126
- @fede-kamel made their first contribution in #50624
- @fxfxfxfxfxfxfxfx made their first contribution in #49139
- @gabriel-peracio made their first contribution in #50183
- @gchinora made their first contribution in #51427
- @guanxingithub made their first contribution in #50589
- @haregali made their first contribution in #50716
- @hsusul made their first contribution in #49613
- @iwannagotobed made their first contribution in #51664
- @jacobzhang22 made their first contribution in #38771
- @jairitAge made their first contribution in #51100
- @jamesETsmith made their first contribution in #51216
- @jimmy-adams made their first contribution in #47972
- @jyan-R made their first contribution in #52311
- @Kaif10 made their first contribution in #52528
- @karen-sy made their first contribution in #48048
- @Lin-z-w made their first contribution in #48069
- @liushujia122 made their first contribution in #51780
- @Luosuu made their first contribution in #48789
- @mispa-ms made their first contribution in #52419
- @mkhazraee made their first contribution in #50321
- @NVShreyas made their first contribution in #50911
- @pavelzak made their first contribution in #52502
- @positive666 made their first contribution in #52329
- @rajfirke made their first contribution in #51573
- @Rapisurazurite made their first contribution in #50950
- @rchalamala made their first contribution in #50487
- @RobbieJ made their first contribution in #49328
- @ruirui6946 made their first contribution in #52098
- @RyanJHamby made their first contribution in #48420
- @samuelkim7 made their first contribution in #50333
- @shanewidanagama made their first contribution in #51901
- @shikamd123 made their first contribution in #45043
- @SilenNaihin made their first contribution in #39935
- @skysnow2001 made their first contribution in #43615
- @Sundaresan-G made their first contribution in #50526
- @syedalijaseem made their first contribution in #47692
- @taking-lying-flat made their first contribution in #51391
- @tandixit95 made their first contribution in #50462
- @Tejas-Raj01 made their first contribution in #49206
- @theminghuang made their first contribution in #51482
- @tobymao made their first contribution in #51318
- @TrainToGPB made their first contribution in #50958
- @tzulingk made their first contribution in #49230
- @UgaTheDev made their first contribution in #51627
- @varoudis made their first contribution in #50404
- @Vegetog made their first contribution in #49876
- @vitamin-chaos made their first contribution in #47106
- @wangxian001 made their first contribution in #50276
- @WillZZZy made their first contribution in #46747
- @xiaopusun made their first contribution in #51495
- @xijiaat made their first contribution in #50693
- @xudonlyu made their first contribution in #51682
- @yifjiang made their first contribution in #51218
- @yu-xin-c made their first contribution in #51652
- @zcxGGmu made their first contribution in #50746
- @ziqifan617 made their first contribution in #51879
- @zobinHuang made their first contribution in #52164
Contributors
@yewentao256, @AndreasKaratzas, @njhill, @mgoin, @aoshen02, @khluu, @hmellor, @stefankoncarevic, @taneem-ibrahim, @LucasWilkinson, @chaunceyjiang, @jikunshang, @zufangzhu, @bigPYJ1151, @NickLucche, @askliar, @fxmarty-amd, @Rohan138, @gty111, @zhenwei-intel, @Isotr0py, @jeejeelee, @ZJY0516, @wangxiyuan, @BugenZhao, @yma11, @zyongye, @DarkLight1337, @jperezdealgaba, @zhou9402, @connorcarpenter15, @kliuae, @lucifer1004, @Alex-ai-future, @aarushjain29, @TheEpicDolphin, @chaojun-zhang, @pmanczak, @WoosukKwon, @BabyDrangoner, @noooop, @almogtavor, @hongxiayang, @fuscof-ibm, @gau-nernst, @zxd1997066, @tlrmchlsmth, @music-dino, @zexplorerhj, @S1ro1, @jdebache, @sfeng33, @mayuyuace, @ganeshr10, @benchislett, @tzulingk, @gcanlin, @ivanium, @divakar-amd, @R3hankhan123, @sagearc, @vhagor, @ronensc, @micah-wil, @qyYue1389, @vanshbhatia-amd, @hao-aaron, @chengy-sysu, @elvircrn, @taking-lying-flat, @omerpaz95, @Etelis, @vllmellm, @frank-suwen, @KernelClint, @Fangzhou-Ai, @louie-tsai, @simondanielsson, @maxyanghu, @dmai-afk, @KurodaKanbei, @ziqifan617, @bastefaniak, @ECMGit, @haregali, @cmiyai, @fede-kamel, @drakosha, @vineethsaivs, @zcxGGmu, @TQCB, @skysnow2001, @shenoyvvarun, @karen-sy, @fattchris, @RyanJHamby, @shikamd123, @namgyu-youn, @zzt93, @abmfy, @reidliu41, @Rapisurazurite, @tandixit95, @mganczarenko, @yimdev, @anujbolewar, @LiuYinfeng01, @lk-chen, @NVShreyas, @huangzhilin-hzl, @varoudis, @Yejing-Lai, @mkhazraee, @jzakrzew, @TrainToGPB, @waynehacking8, @zixi-qi, @Sundaresan-G, @mindungil, @bitborne, @Wauplin, @jacobzhang22, @zhewenl, @bnellnm, @pisceskkk, @Lin-z-w, @gabriel-peracio, @SilenNaihin, @baodii, @YunzhuLu, @xwu-intel, @BWAAEEEK, @thisjiang, @maobaolong, @anhtra3889, @JaredforReal, @lvhan028, @xiaolong-intel, @andyxning, @cleonard530, @gnovack, @MatthewBonanni, @wangxian001, @lengrongfu, @Tejas-Raj01, @simon-mo, @vitamin-chaos, @arpera, @jairitAge, @jimmy-adams, @ILikeIneine, @woosebastian, @haic0, @edwinlim0919, @fcui-amd, @jhu960213, @jinzhen-lin, @coltonottley, @walterbm, @meenchen, @matteso1, @djramic, @gchinora, @davidjpyu, @tianmu-li, @xiaopusun, @majunze2001, @Vegetog, @puririshi98, @janeyx99, @RobbieJ, @oonyshch, @thegoldenflow, @Srinivasoo7, @fatday, @acheamponge, @efschu, @rajfirke, @fanxingran, @xudonlyu, @lcskrishna, @xijiaat, @GirasoleY, @d4l3k, @samuelkim7, @tarukumar, @acmore, @theminghuang, @khushali9, @wzhao18, @Priyjain-amd, @yiz-liu, @lkm2835, @dmholtz, @Dao007forever, @liushujia122, @LopezCastroRoberto, @UgaTheDev, @tuukkjs, @aditi-amd, @guan404ming, @yiliu30, @zou3519, @Luosuu, @JoursBleu, @varun-sundar-rabindranath, @mpashkovskii, @yu-xin-c, @WillZZZy, @vrdn-23, @xyang16, @ccrhx4, @tanpinsiang, @russellb, @fxfxfxfxfxfxfxfx, @afriedri, @yifjiang, @Akashcodes732, @HF-001, @ovidiusm, @arthurgao2003, @TomerBN-Nvidia, @hotTea123, @vx120, @bohnstingl, @qwerqwerqwe8688-jpg, @jasonozuzu-cohere, @vineetatiwari27, @ruirui6946, @linitra24, @syedalijaseem, @nickus, @yzong-rh, @s3woz, @jhaotingc, @lukealonso, @Jie-Fang, @kzwrime, @xianbaoqian, @velonica0, @ccaadaro, @yisustc, @fangchenli, @iwannagotobed, @zobinHuang, @rchalamala, @shanjiaz, @jamesETsmith, @stacyroberts, @guanxingithub, @biswapanda, @shanewidanagama, @UranusSeven, @hsusul, @tobymao, @mispa-ms, @jeffreywang88, @SayHelloToWorld, @jyan-R, @oops-oom, @shantipriya-amd, @andakai, @akii96, @shen-shanshan, @Kaif10, @yitingdc, @positive666, @pavelzak, @SubSir, @ywang96
@ Every self-hosted release, weekly
One Monday email with the week's stable releases across the directory.
>_ Models vLLM runs
Popular open-weight models with vLLM support, with VRAM needs and licence at a glance.
More in LLM Inference
vllm-qwenSelf-hosted OpenAI-compatible inference server running Qwen3.6-27B BF16 on AMD Ryzen AI Max+ 395 with vision support, 256K context, and separated reasoning output.
WhisperLive---CPUSelf-hosted, open-source real-time Whisper transcription server with faster-whisper, TensorRT and OpenVINO backends and an OpenAI-compatible REST API.
YALSYALS is a self-hosted, open-source OpenAI-compatible API server for GGUF models via llama.cpp, with continuous batching and JSON schema support.
AGmindAGMind is a self-hosted, open-source installer deploying a private RAG stack on NVIDIA DGX Spark in ~30 minutes via an interactive wizard.
aisuiteDocker Compose suite bundling N8N, Flowise, Langflow, LocalAI, Ollama, VLLM, Qdrant, ChromaDB, and Stable Diffusion behind Traefik with auto HTTPS.
aixclSelf-hosted platform for running LLMs locally with Vault secrets, GPG-signed commits, and rootless Podman. Includes Open WebUI, pgAdmin, Grafana, and BYO app framework.
aphrodite-engineOpen-source, self-hosted LLM inference engine for Hugging Face models: continuous batching, paged KV cache, quantization, speculative decoding, OpenAI-compatible serving.
ARenoAReno is an open-source, self-hosted toolkit for RL post-training of LLMs on a single node with GSPO, GRPO, SFT, DPO, agentic RL, LoRA/QLoRA, and OpenAI serving.
Atomic ChatSelf-hosted desktop/mobile app for running LLMs locally with OpenAI-compatible API. 1000+ models, multiple engines, MCP, agent launching.
distributed-llamaDistributed LLM inference tool connecting home devices into clusters using tensor parallelism over Ethernet for accelerated performance.
exllamaOpen-source Python/C++/CUDA runner for 4-bit GPTQ-quantized Llama models on consumer NVIDIA GPUs, with a self-hosted chat web UI you can build and run in Docker.
exoSelf-hosted, open-source AI cluster that links your Macs and Linux boxes to run large LLMs locally via MLX, with Thunderbolt 5 RDMA and an OpenAI/Ollama-compatible API.
0 comments