Self-host sglang
v0.5.23High-performance LLM and multimodal inference framework
- Stars
- 38 926
- Port
- 30000 (HTTP API server)
lmsysorg/sglangWhat is sglang?
SGLang is a high-performance serving framework for large language models, vision-language models, and diffusion models. It targets production deployments from a single GPU to distributed clusters, delivering low-latency and high-throughput inference. The framework supports many open models including Llama, Qwen, DeepSeek, GLM, and Mistral, and runs on diverse hardware such as NVIDIA GPUs, AMD GPUs, Google TPUs, Intel CPUs/GPUs, Apple Silicon, Huawei Ascend NPUs, and Moore Threads GPUs. SGLang provides OpenAI-compatible API endpoints, advanced optimizations like disaggregated prefill/decode, speculative decoding, tensor/expert/data parallelism, prefix caching, and optimized GPU kernels. Deployment options include Docker or Python installation with uv. The project is hosted by LMSYS, a non-profit open-source organization, and is licensed under Apache 2.0.
sglang alternatives
Turbo-LLMSelf-hosted local LLM platform running any engine with auto-tuning, real measured tokens/sec, and OpenAI/Anthropic APIs for Claude Code.CompareTurbo-LLM with sglang
llmakerllmaker is a self-hosted, open-source CLI that provisions and manages complete LLM stacks on your own hardware with a single command.Comparellmaker with sglang
difyDify is a self-hosted, open-source LLM app platform with visual workflows, RAG pipelines, agents, and model management for taking AI apps from prototype to production.Comparedify with sglang
heliconeHelicone is a self-hosted, open-source LLM observability platform and AI gateway for 100+ models with cost tracking, agent tracing, prompt management, and a playground.Comparehelicone with sglang
llamactlSelf-hosted control plane that manages llama.cpp, MLX, and vLLM inference servers through a web dashboard with a unified OpenAI-compatible API.Comparellamactl with sglang
LLMKubeOpen-source Kubernetes operator running LLMs on NVIDIA, AMD, Intel, and Apple Silicon with pluggable runtimes and hybrid routing.CompareLLMKube with sglang
Text Generation WebUIAn open-source, self-hosted LLM chat app you run on your own machine: vision input, tool-calling, LoRA fine-tuning, and an OpenAI/Anthropic-compatible API.CompareText Generation WebUI with sglang
GaiaNet NodeOpen-source, self-hosted GaiaNet node: pick any Hugging Face LLM, add a RAG knowledge base, serve an OpenAI-compatible API and web chat UI.CompareGaiaNet Node with sglang
Should you self-host sglang?
sglang scores 85/100 on selfhost.directory (how scoring works) and ranks #4 of 25 in LLM Inference. It has 38 926 GitHub stars, +1 874 in the last 30 days.
It shipped 18 stable releases in the past year. The latest, v0.5.23, came out on 30 October 2026. New versions land roughly every 2 weeks.
Setup is advanced: plan for several services and some ongoing maintenance. An official Docker image is available. It is Apache-2.0-licensed and written mostly in Python.
Its closest alternative on selfhost.directory is Ollama (94/100, 187 151 stars). It scores higher overall, so compare the two before you pick. See sglang vs Ollama side by side.
Features
- Broad hardware support
- Context parallelism
- Data parallel attention
- Disaggregated prefill/decode
- Expert parallelism
- LoRA support
- Multi-modal support
- OpenAI Compatible API
- Prefix caching
- Quantization Support
- Speculative Decoding
- Tensor Parallelism
sglang README
Docs | Cookbook | Website | Blog | Slack
SGLang is an open-source inference framework for large language, vision-language, and diffusion models, optimized for agentic workloads, large-scale serving, and RL rollouts. SGLang Diffusion is its built-in image and video generation engine, included in this repository and the sglang Python package.
How to install sglang
Install sglang with Docker
docker pull lmsysorg/sglang:latest
docker run --rm --gpus all --ipc=host
-p 30000:30000
lmsysorg/sglang:latest
sglang serve MODEL_PATH --host 0.0.0.0 --port 30000
Requires NVIDIA Container Toolkit and CUDA 13-compatible driver
Install sglang with Python
uv venv --python 3.12 source .venv/bin/activate uv pip install --prerelease=allow sglang sglang serve MODEL_PATH --host 0.0.0.0 --port 30000
Requires Python 3.12+ and uv package manager
sglang releases & version history
Highlights
1007 PRs from 257 contributors.
New models
| Model | Type | Cookbook |
|---|---|---|
| Muse Spark Open | LLM / VLM | link |
| Beam-1.0 | LLM / VLM | link |
| MiniMax-M3.1-Flash | LLM / VLM | link |
| MiniCPM-V 4.7 | LLM / VLM | link |
| Kev | LLM / VLM | link |
| Cosmos3-Nano-Sim (Bimanual, Transfer, Auto) | Diffusion | link |
| FastH3 Omni | Diffusion | link |
| FLUX.3 Image Klein | Diffusion | link |
| HY-Nomas | Diffusion | link |
| MOSS-Image | Diffusion | link |
What's new
- DeepSeek-V4.1 on GB300 gets up to 1.5x more prefill throughput under load and about 28% lower first-token latency on 32K to 64K prompts (#44403, #45059, #45107).
- SGLang low-latency mode: the opt-in NVIDIA megakernel and AMD monokernel plugins run DeepSeek-V4.1-Flash decode as fused persistent kernels, for up to 44% faster single-user decode on GB300 and 75% on MI350X (#44945, #45450, #44353, #44621).
- MoE models such as DeepSeek-V4.1-Flash, GLM-5.3-Flash and gpt-oss reach the first token 30% to 58% sooner on 8K prompts on H200 (#43363).
- Paged Experts (opt-in) runs MoE models larger than GPU memory on a single consumer GPU, up to 30x faster than CPU offload (#42557, #45787).
- SGLang now runs on Google TPU7x through SGLang-TorchTPU, a PyTorch-native plugin for TPU.
- You can now serve LoRA adapters on quantized MoE models (FP8 and NVFP4) with a new opt-in LoRA engine (#45892).
- DeepSeek-style MLA models running decode context parallelism can now keep their prefix cache in HiCache file-based L3 storage (#41187).
To upgrade
uv pip install --prerelease=allow sglang==0.5.23
| Platform | Docker |
|---|---|
| NVIDIA (CUDA 13) | docker pull lmsysorg/sglang:v0.5.23 |
| AMD MI35x | docker pull lmsysorg/sglang:v0.5.23-rocm10-mi35x |
| AMD MI30x | docker pull lmsysorg/sglang:v0.5.23-rocm10-mi30x |
| Intel GPU | docker pull lmsysorg/sglang:v0.5.23-xpu |
| Intel CPU | docker pull lmsysorg/sglang:v0.5.23-xeon |
| Google TPU7x (TorchTPU) | docker pull lmsysorg/sglang:v0.1.0-tpu |
Full Release Notes
Speculative Decoding
- [Feat] speculative: Seed rejection-sampling draft proposals for deterministic inference: #33395
- [Fix] observability: Correct speculative FPM timing and decode lengths: #38711
- [Perf] spec: Let the FlashAttention backend capture the draft-extend CUDA graph (about 1.5x decode throughput at batch size 1 with a single-layer MoE drafter): #40762
- [Perf] spec-decoding: fuse simulated speculative acceptance tensor preparation: #41169
- [Fix] dsa: Fix ragged seed indices before MTP draft decode: #42565
- [Spec] Fix DFLASH verify mask policy for wrapped attention backends: #43054
- [Feat] spec: Capture aux hidden states on the target without a speculative draft: #43449
- [Fix] DSv4.1: Size the encoder-replay SWA pool by the DSpark draft's stages, and keep more prefix tails: #43502
- [Fix] dsa-eagle-topk: Copy token keys and scales when relocating index cache: #44564
- [Fix] deepseek-v4.1: Keep the DSpark draft SWA out of the radix tree: #44673
- [Perf] speculative: Fuse draft write-window gathering and KV translation: #44972
- [Perf] attention: Capture translated FA4 verify metadata for DSPARK: #44973
- [Fix] deepseek-v4.1: Start the PD decode draft SWA window at the prefill draft valid-from: #45037
- [Fix] spec: Remove
--speculative-skip-dp-mlp-sync: #45043 - [Feat] spec: Let out-of-tree platforms provide their own speculative worker: #45144
- [Feat] spec: Let out-of-tree platforms provide EAGLE draft graph runners: #45145
- [Fix] mamba: Skip zero-width conv pools in the MTP verify scatter: #45212
- [Fix] deepseek-v4: Keep the draft MoE out of the target's expert placement: #45299
- [Perf] [NVIDIA] dspark: Move the host work between DSv4.1 decode graphs into the graphs for the megakernel path: #45450
- [Feat] HiCache: Support fused DFLASH KV in unified hybrid SWA pools: #45700
- [Perf] gemma4: Capture single-step frozen-KV drafts (Gemma 4 E2B median TPOT 33% lower at concurrency 1 on H200): #45771
- [Perf] dsv41: Trim per-step DSpark decode overhead: #46176
- [Feat] spec: Support CUDA attention DP for DFlash and DFlash2: #46330
- [Perf] [NVIDIA] kda: Narrower V tiles for the KDA spec-verify recurrence on SM107: #46465
- [Perf] [NVIDIA] kimi-k3: Route draft/dense bf16 GEMMs to TGV and narrow the MLA output gate vector: #46468
Breakable CUDA Graph
- [Perf] prefill-cuda-graph: Make prefill shared-read completion explicit and default-on: #37958
- [Perf] deepseek-v4.1: Replay the breakable prefill CUDA graph at any context length: #43733
- [CP] Automatically enable BCG for supported prefill CP configurations: #44022
- [Fix] DSv4.1: Fix BCG image handling and add multimodal bounded replay coverage: #44131
- [Fix] Qwen3.8next: Stop retaining per-size PLE outputs in breakable prefill CUDA graphs: #44150
- [Fix] cuda_graph: Wait for in-flight layer-wise cache loads before prefill graph replay: #44220
- [DSv4.1] Support TRT-LLM decoder bounded replay with BCG: #44326
- [Perf] deepseek-v4.1: Run encoder SWA bounded replay in the breakable prefill graph: #44930
- [Perf] deepseek-v4.1: Run decoder replay tail layers from CUDA graphs on eager prefill steps (saturated 8K prefill throughput +33% on GB300 at TP4): #45059
- [Perf] deepseek-v4.1: Run decoder replay tail graphs in prefill-graph steps and under DP attention (saturated prefill +37% at TP4 and +50% with DP4 attention): #45107
- [Fix] glm-5.2: Record side-stream work right before its join to keep CUDA-graph replay on one stream: #45349
- [Fix] [NVIDIA] deepseek-v4.1: Fix the breakable prefill CUDA graph in the late-layer tail on SM90: #45407
- [Perf] cuda-graph: Capture FlashInfer KDA prefill with context parallelism: #45473
- [Perf] cuda-graph: Raise MLA prefill capture default to chunked size capped at 8192: #45565
- [Perf] dsv4: Let the decoder tail leave the prefill graph under DP attention with MegaMoE: #45581
- [Fix] prefill-graph: Keep every DP rank eager when the replay tail captures behind rank barriers: #45881
- [Perf] dsv4: Replay the V4.1 prefill graph past 16K contexts on SM90: #45941
- [Fix] dp-attention: Align prefill graph buckets and decode padding across DP ranks: #46138
- [Perf] deepep_v2: Reduce masked GEMM buffer memory during CUDA Graph capture: #46309
Attention Backends
- [Perf] fmha_v2: use non-interleaved paged KV input for trtllm sm90/120 GQA prefill: #32272
- [Fix] attention: Honor selected FlashAttention implementation: #33007
- fix: FA4 paged-KV zero-size page-table registers on SM90: #35757
- Fix overflow in legacy and v1 DSA top-k selectors: #37625
- [Fix] dsa: Make the fused top-k exact on an overflowing threshold bin: #37941
- [Fix] DSA: skip-topk pipeline handoff for owed outputs: #43832
- [Fix] attention: Opt FlashMLA scheduler into dynamic shared memory: #44580
- [Chore] attention: Remove the tokenspeed_mla backend and dependency: #44803
- [Feat] attention: Add the MSA-v1 sparse backend and NVFP4 KV pool: #46141
- [Fix] dsa: Preserve token-first sparse attention outputs: #46424
MoE & Expert Parallelism
- [Fix] moe: Fix flashinfer TRT-LLM BF16 expert weight reload on refit: #33743
- [Fix] eplb: Fix Qwen3 MoE output corruption after EPLB rebalance: #39580
- [Fix] moe: Preserve negative routing sentinel in EP mapping: #40498
- [Fix] moe_sum_reduce: gate the bf16 vectorized path on hidden % 16: #42837
- [Perf] moe: Extend FlashInfer CUTLASS MoE autotune coverage to prefill chunks (TTFT 30% to 58% lower for 8K prompts on seven MoE models on H200): #43363
- [Feat] FlashInfer: Add NVFP4 W4A16 and SiTU support to MegaMoE: #44036
- [Fix] moe: gated NVFP4 MoE intermediate alignment for FlashInfer TRTLLM: #44138
- [Fix] moe: Map expert ids for layers whose runner needs local ids under EP: #44480
- [Fix] megamoe: Respect request-pool limits in decode validation: #44816
- [Fix] megamoe: Support EPLB rebalance on SM100: #44847
- [Fix] moe: Count routed_scaling_factor once when top-k folds it: #44901
- [Fix] moe: Renormalize per-rank shared-slot routing over routed experts only: #44904
- [Fix] [NVIDIA] modelopt-nvfp4: Keep MoE global scales in gate/up order: #45289
- [Perf] moe: Preserve Marlin workspace and route-count budgets: #45462
- [Perf] moe: Add H200 block-FP8 Triton configurations: #45591
- [Fix] gguf: Preserve MoE routing, weighting and allocation budgets: #45916
- [Fix] moe: Preserve CUTLASS routing metadata and execution budgets: #45917
Quantization
- [Fix] quantization: Preserve BF16 NaNs in per-token FP8 quantization: #42689
- [Fix] compressed-tensors: Serve NVFP4 W4A4 linears via FP4 Marlin when --fp4-gemm-backend marlin is set on SM100: #44213
- [Fix] nvfp4: Own streamed tensors while waiting for paired weights: #44500
- [Fix] [NVIDIA] modelopt-nvfp4: Read w13 global scales in the TRT-LLM [up, gate] order: #44858
- [Perf] mlp: Fuse dense SwiGLU with block-FP8 quantization on decode (single-request decode TPOT 1.9% lower on Qwen3-8B-FP8 on B200): #45219
- [Fix] modelopt: Correct NVFP4 TRT-LLM clamps for untied gate/up scales: #45386
- [Fix] [SM100] quantization: Honor
deep_gemmfor 32x32 block-FP8 linears: #45585 - [Feat] quantization: Add the MiniMax-M3.1 NVFP4 MoE method: #46140
Parallelism & Disaggregation
- [4/N] elastic-ep: Scale at DP-attention replica granularity: #33728
- [Fix] hybrid-SSM: DP attention failures at low concurrency: #34535
- [Feat] KVShard: PD transfer and e2e test: #38473
- [Perf] pd: Stage decode-side KV preallocation lengths through pinned memory: #38955
- [Qwen3.8 CP 4/4] Prefill context parallel E2E support of Qwen3.8: #39724
- [Fix] disaggregation: Clean-up and optimize the KV checksum implementation: #39729
- [Feat] KVShard: Support kv cache sharding for MTP: #40929
- [Fix] epd: Accept tokens-in requests on scheduler-side transfer backends: #41376
- [Fix] PD: retraction restores racing the in-flight decode forward: #42702
- [Perf] [NVIDIA] custom-all-reduce: Add 2shot_push all-reduce and retune SM100 TP4/TP8: #42811
- [Feat] KVShard: Integrate GLM DSA/MTP with PD and CPU HiCache: #42973
- [Feat] cp: Enable GLM-5.3 Flash interleave BCG with batched prefill and DFlash2: #43182
- [Fix] nccl: Disable graph buffer registration whenever a captured graph can hold a collective: #43216
- [Feat] NIXL: Support per-GPU UCX device selection: #43482
- [Fix] disaggregation: Reuse startup decode graphs across PD role switches: #43760
- [Feat] glm5.3flash: Support padded CP BCG and prefix replay with PD/DFlash2: #43774
- [Perf] CP: QSA Context Parallel Optimization (26.5% lower QSA prefill latency at 8K with CP4): #43792
- [Feat] dsv4.1: Support encoder SWA bounded replay with CUDA DP attention: #43813
- [Perf] KVShard: 0SM KV Cache Allgather for KV Cache Sharding: #44029
- [PD] Send DSA indexer state only past decode_prefix_len: #44140
- [Fix] pd: Allow runtime role switching with DP and DP attention: #44215
- [Feat] KVShard: Add attention and indexer support for page-sharded DSA caches: #44280
- [Fix] scheduler: Keep PD decode non-idle while deferred KV releases are held: #44386
- [Fix] [NVIDIA] allreduce-fusion: Decline MNNVL fusion across NVLink cliques: #44387
- [Fix] pp: Order output send/recv by rank parity to fix cross-node hang: #44453
- [Perf] pd: Copy prebuilt decode batch inputs to the GPU without host syncs: #44598
- [Feat] disaggregation: Let a KV transfer backend skip the checksum for an incomparable peer: #44605
- [Perf] dp-attention: Exchange the scheduler's MLP sync rows with gather + broadcast on gloo: #44622
- [Perf] distributed: Cache the eager launch of custom all-reduce v2 per size: #44642
- [Perf] NIXL: prepared dlists for hybrid-model state transfer: #44740
- [Feat] deepseek-v4.1: Run encoder SWA bounded replay on PD prefill workers: #44952
- [Perf] deepseek-v4.1: Default decoder SWA bounded replay on for both PD roles on CUDA: #45035
- [Perf] deepseek-v4.1: Default PD to MegaAttention, with the KV layout in the PD handshake: #45075
- [Feat] mooncake: Enable the failed-session probe by default with backoff and idle pruning: #45338
- [Fix] disaggregation: Keep a requeued optimistic prefill request's multimodal inputs: #45440
- [Fix] disaggregation: Synchronize deferred decode KV release across TP ranks: #45453
- [Perf] pd: Send prefill KV chunks without a host sync on the scheduler thread: #45492
- [Fix] NIXL: Preserve explicit UCX network device selection: #45722
- [Fix] disaggregation: Restart a lost prefill's heartbeat miss count: #45922
- [Fix] disaggregation: Poison the deferred abort ack when a NIXL or Mori write fails: #45973
- [Feat] layer-boundary: Fuse TP all-reduce, residual add and RMSNorm on the push plane: #46137
- [Perf] dsa: Overlap CP KV gather with indexer computation: #46179
- [Perf] pd: Apply the prefill WAR barrier after the previous result is processed: #46220
- [Feat] dcp: Replicate a block-FP8 q_b_proj under --dcp-replicate-q-proj: #46359
- [Fix] [NVIDIA] dist: Detect MNNVL fabric devices by NVML fabric state, not only by GB200/GB300 name (Kimi-K3 on two-node TP8/DCP8 decodes at 485 instead of 379 tok/s): #46463
Scheduler & Runtime
- [Fix] Fix multi-tokenizer BatchEmbeddingOutput repack dropping fields: #31892
- [Fix] detokenizer router: Route by rid without tokenizer worker ipcs and broadcast logging config: #33715
- perf: fix overlap scheduling for NVIDIA Confidential Computing(CC) on Blackwell: #36810
- [Fix] tracing: Fix OTel tracing context lost in multi-tokenizer router processes: #38211
- [Perf] weight-sync: Allocate packed receive buffers from metadata: #39264
- [Feat] bench: support --tokenize-prompt for generated-shared-prefix: #40900
- [Fix] Keep rope_theta when --json-model-override-args overrides rope_scaling: #43067
- [Feat] metrics: Expose multi-detokenizer router CPU usage: #43323
- [Perf] Cache msgspec struct fields in the batch splitter: #43520
- [Feat] runtime: Include FlashInfer’s default tactic in startup tuning.: #43927
- [Feat] bench: report goodput and SLO attainment in bench_serving: #43967
- [Feat] scheduler: Gate --prefill-decode-interval on running batch size: #44001
- [Perf] Bind scheduler CPUs to the GPU's NUMA node when memory policy is not permitted: #44023
- [Fix] offloader: Preserve cached weights during CPU offload: #44139
- [Perf] detokenizer: Group return IPC sends by tokenizer worker (+2.1% output throughput on DeepSeek-V4-Pro): #44288
- [Fix] scheduler: Drop finished requests from the running batch before every admission: #44351
- [Fix] scheduler: Count a request's held req_to_token row once at admission: #44401
- [Fix] imports: Bind FlashInfer lazily at four import sites so the server starts without it: #44430
- [Perf] [NVIDIA] scheduler: Pageable host staging and inline hidden-state/DSpark readback under Confidential Computing: #44568
- [Perf] request_logger: skip per-token watermark redaction walk: #44692
- [Feat] profiler: Add the forward pass id to detailed step span labels: #44775
- [Fix] srt: Apply --gc-threshold in every server process and freeze every tokenizer worker: #44785
- [Fix] scheduler: Exclude accepted storage-tier hits from the waiting prefill load: #45158
- [Perf] detokenizer: Add SGLANG_BATCH_TOKENIZER_OUTPUTS to opt out of grouped return sends: #45280
- [Feat] platforms: Let out-of-tree platforms bind group devices and custom ops: #45897
- [Fix] weight_checker: Make the checksum device-agnostic: #45918
- [Feat] metrics: Export prealloc_{full,swa}_token_usage for decode requests with a pending KV transfer: #45970
- [Feat] benchmark: Add --input-len-list to bench_one_batch_server for per-request input lengths: #45982
- [Fix] scheduler: Keep long prefill steps in the busy counter and exclude engine pauses: #46127
- [Fix] request_logger: Skip watermark redaction when watermarking is disabled: #46207
Sampling
- [Perf] logits: Write lm_head logits straight into the fp32 logits buffer: #45631
- [Fix] grammar: Give grammar accept failures an error status: #46049
- [Perf] logits: Cast bf16 logits into the fp32 buffer with a Triton kernel: #46167
HiCache & Radix Cache
- [Fix] unified-memory: Bound joint reclaim and preserve cached prefixes: #39294
- [HiCache] Add the page_unified host KV layout and the unified L3 key: #39727
- [Feat] hicache: Expose the Mooncake L3 object-key layout for external KV routers: #40467
- [Fix] hicache: Fail fast on a null pointer from the Mooncake host allocator: #40565
- [Feat] unified-cache: Support out-of-tree external linkers: #40595
- [mem_cache][PD] Fix decode radix cache ownership and allow EAGLE/EAGLE3 with decode radix cache: #40857
- [Feat] hicache: Support MLA DCP with HiCache L3(file/mooncake): #41187
- [Feat] unified_cache: Support hybrid SWA + Mamba models in the Mooncake direct linker: #41343
- [Fix] hicache: Key MLA Mamba state per TP rank in the file backend: #41907
- [Fix] HiCache: stale host KV from write-through copies racing the in-flight forward: #42701
- [Fix] unified-memory: enabling trtllm_mha with decode graphs and speculative decoding: #42930
- [Fix] VLM: harden Kimi image cache identity across preprocessing and radix hashing: #43043
- [Fix] swa: Retain a whole window behind insertion pages: #43168
- [Perf] unified-memory: Free SWA segments without a host sync: #43170
- [Feat] unified-memory: Support asymmetric KV rows: #43325
- [Feat] unified-memory: Fuse drafts with sliding-window and recurrent-state layers: #43326
- [Fix] hicache: Keep unified memory off the direct page_first_direct copies: #43333
- [Perf] hicache: Unify PP prefetch hit queries and buffer allocation: #43344
- [Fix] unified_cache: Prevent misaligned Mamba checkpoints after speculative truncation: #43387
- [Fix] hicache/pd: Promise prefill only the L3 span the decode prefetch takes: #43548
- [Feat] hicache: Export host Mamba pool occupancy next to host KV: #43727
- [Fix] hicache: Key MLA Mamba state per TP rank in the SeaweedFS backend: #43816
- [Fix] hicache: Key MLA Mamba state per TP rank in the HF3FS backend: #43817
- [HiCache] Buffer-only L3 for an auxiliary SWA window; load back only missing window rows: #43932
- [Perf] hicache: Parallel THP-backed host pool allocation (opt-in) (host pool allocation about 4x faster, about 140 s saved per start): #43978
- [Fix] PD: Skip the cached-prefix early send in batches that start a HiCache load-back: #44081
- [Fix] unified-memory: Avoid gather scratch during Mamba and Conv compaction: #44097
- [Feat] hicache: Add opt-in tail-only SWA backups in buffer mode: #44173
- [Perf] unified-memory: Prewarm the compaction move at startup and copy whole page envelopes by page id: #44592
- [Fix] hicache: Bound the storage prefetch timeout from the first blocked admission poll: #44614
- [Fix] cache: Support DeepSeek-V4.1 external linker pools: #44623
- [Fix] hicache: Order write-back host copies after the forward that produced the KV: #44800
- [Feat] server-args: Add --enable-hicache alias for --enable-hierarchical-cache: #44849
- [Fix] cache: Accept packed draft layer mappings in the UMBP linker: #44929
- [Perf] unified-memory: Reduce allocation and SWA retirement overhead: #44970
- [Feat] unified_cache: Support FULL + MAMBA MLA models (Kimi-K3) in the direct external linker: #44977
- [Fix] hicache: Count the EAGLE bigram boundary token in PD decode restores: #45076
- [Fix] hisparse: Copy DeepSeek-V4 eager backups token by token: #45104
- [Fix] hisparse: Keep PD destination host pages until the decode KV release: #45108
- [Perf] unified-memory: Allocate free pages from each sub-pool's own end first: #45222
- [Fix] mem_cache: Skip linker L3 lookup on PD decode: #45538
- [Feat] disaggregation: Verify the KV checksum under a decode-side radix cache: #46423
LoRA
- [KV events] Publish opaque LoRA block IDs and adapter names: #42817
- [Fix] openai: distinguish s3 served-model-name from LoRA adapter syntax: #44450
- [Fix] lora: Preserve MoE projection identity and routing lifetimes: #45681
- [Feat] lora: Add plan-driven dense and MoE LoRA engines: #45892
Multimodal
- [Fix] inkling: Return 400 instead of 500 for unfetchable/undecodable multimodal input: #40899
- [Feat] runtime: Support tokenized multimodal generation: #42833
- [Fix] glm4v: Clamp request video budgets to the deployment limit: #43871
- [Perf] session: Cache multimodal prompt token counts across turns: #44112
- [Fix] multimodal: Preserve caller hashes and offsets for precomputed inputs: #44160
- [Fix] nemotron_vl: Preserve tokenized prompts during media expansion: #44184
- [Fix] nemotron_vl: Pad visual spans using processor offsets: #44581
- [Fix] multimodal: Clip pad spans to a truncated prompt: #46638
Model Support & Optimizations
- [Fix] deepseek-v4: Fix DeepSeek-V4 non-EP TBO for attention TP > 1: #33250
- [Fix] dLLM: Preserve prompt tails and literal mask tokens across prefix reuse: #39592
- [Fix] dsv4: Skip DP-attention pad rows in the engram hasher and low-ratio sources: #40424
- [Fix] dLLM: Size decode graphs to the default max running requests: #40941
- [Fix] deepseek-v4.1: Match KV capacity to physical layouts: #41275
- [Fix] deepseek-v4.1: Keep the full SWA window for short bounded-replay extends: #41303
- [DeepSeek V4.1] Use DeepSelect for Hopper full top-k decode.: #41561
- [Fix] deepseek-v4: Refresh attn_sink after loading weights: #41630
- [Feat] MiniCPM-V-4.7: add model support: #41993
- [Fix] qwen3.8: Fix batch-invariant kernels for Qwen3.8 Flash-Next: #42087
- [Perf] deepseek-v4.1: Skip wasted layer 20 work on earlier prompt tokens under decoder bounded replay: #42901
- [Perf] dsv4: Vectorize paged KV cache dequantization: #42929
- Fuse MiMo verification RoPE and KV cache writes (E2E throughput +4.6%): #43106
- [DSv4.1] Fold encoder-SWA replay into the extend forward: #43166
- [Perf] gdn: Select FlashInfer prefill for page-major pools: #43171
- [Perf] dsv41: Integrate DeepGEMM MegaGate routing: #43262
- [Fix] mamba: Release consumed lazy Mamba prefix locks before donation retry: #43559
- [Fix] kimi-k3: Preserve pipeline graph bank inputs and bound chunk profiling: #43604
- [Perf] Enable native MXFP4 routing and CuTeDSL fusion for MiMo (decode throughput +11.8% with DFlash): #43639
- Support EPLB for Nemotron-H: #43685
- [Perf] dsa: Route the k-pool indexer top-k through the top-k v2 kernel: #43775
- [Perf] DeepSeek-V4.1: Run the small-batch decode fusions at TP2: #43838
- [Perf] deepseek-v4.1: Integrate FlashMLA MegaAttention with NVFP4 KV: #43922
- [DeepSeek-V4.1] DeepSelect cluster top-k for long rows and bf16 index logits: #44051
- [Fix] glm5_next: Correct num_experts_per_tok default to 8: #44093
- [GLM-5.3-Flash] Drop the NextN layer from the KDA state pool: #44098
- [Perf] [NVIDIA] qsa: Keep sparse decode scratch in FP8 for FP8 KV pools: #44114
- [Fix] DeepSeek-V4.1: Align paged indexer score rows for DeepSelect decode: #44210
- [Perf] DSv4.1: Enable decoder bounded replay by default (about 28% lower first-token latency at 32K and 64K input): #44403
- [Fix] nemotron: Preserve nested Puzzle configs: #44487
- [Feat] nemotron: Port Nemotron 3.5 Super serving changes: #44518
- [Perf] [NVIDIA] muse-spark-open: Fuse pre-expert RMSNorm with static-scale NVFP4 quantization: #44554
- [DSv4.1] Fuse request-window KV copies (median E2E latency about 61% lower with encoder SWA bounded replay on a high-load replay on 4x GB300): #44555
- [Perf] deepseek_v4: size the masked silu-mul-quant grid to the GPU, not the padded capacity: #44618
- [Perf] DeepSeek-V4.1: Raise the SM100 swap-AB decode cap to 40 rows: #44632
- [Feat] kernels: Add muse spark operators and collective GEMM (2/5): #44665
- [Feat] srt: Add muse spark runtime, multimodal support and tests (3/5): #44666
- [Fix] bailing: MoE gate dtype follows config.router_dtype: #44715
- [Perf] DSv4.1: Skip discarded KV rows during window commit (median E2E latency 7% lower on 4x GB300): #44783
- [Fix] engram: Avoid blocking history copies during PD admission: #44787
- [Perf] deepseek_v4: fuse block-FP8 shared experts into MegaMoE for any 32-aligned block size: #44837
- [Perf] deepseek_v4: fold the MegaAttention query layout into wq_b for block-FP8-as-MXFP8 checkpoints: #44838
- [Perf] deepseek_v4: run FlashInfer autotune for block-FP8 models served as MXFP8: #44842
- [Fix] deepseek-v4.1: Serve prompt-logprob requests under decoder SWA bounded replay: #44850
- [Perf] mamba: Fuse strided state updates and replay index translation: #44971
- [Fix] DSv4.1: Support CUDA image requests with encoder bounded replay: #45048
- [Feat] dsv4.1: Return prompt logprobs under encoder SWA bounded replay: #45050
- [Fix] nemotron: Correct weight-export failures and GEMV dispatch: #45082
- [Fix] kimi-k3: Avoid vision attention child-name collisions: #45110
- [Perf] mamba: Copy tracked decode states at HBM bandwidth in one persistent launch: #45213
- [Feat] kernels: Beam model support: #45254
- [Fix] inkling: Keep BF16 LM-head operands with FP32 output: #45335
- [Feat] Inkling: render and parse chat via tml-renderers: #45340
- [Fix] muse_spark: support unified KV memory with TRT-LLM: #45356
- [Feat] systemone: serve Kev pointer head decision models: #45413
- [Fix] mamba: Keep prefill rows' checkpoint tracking through mixed chunked prefill: #45435
- [Perf] scheduler: Plan hybrid Mamba extend batches without host-device syncs: #45472
- [Perf] [KDA] muse_spark: Run the one-token router GEMV on CuTe DSL: #45508
- [Perf] dsv4: Restore sparse prefill for request-window rows (+9.6% throughput per GPU at 32K tokens per rank on GB300): #45573
- [Perf] dsv41: Use tuned MXFP8 tactics at decode sizes: #45630
- [Perf] dsv41: Mega-mHC and the wo_a einsum write MXFP8 for wqkv_a and wo_b: #45632
- [Perf] dsv41: Fuse the q_norm MXFP8 quantize for wq_b at every row count: #45633
- [Perf] [KDA] muse_spark: use BLOCK_M 240 for BF16 MegaMoE prefill shards: #45637
- [Perf] QSA: Optimize the prefill indexer gather: #45752
- [Perf] QSA: Optimize prefill indexer in long cached prefixes: #45791
- [Perf] [NVIDIA] deepseek-v4.1: Align BF16 prefill indexer score width: #45794
- [Perf] dsv4: Score the V4.1 index logits in bf16 on SM100: #45880
- [Perf] dsv41: Serve token-interleaved V4.1 KV caches with sgl-flash-mla: #45893
- [Perf] dsv4: Use the fused MXFP8 producers in eager prefill: #45903
- [Perf] dsv4: Optionally stage V4.1 engram rows from the shared host table on the scheduler: #45921
- [Feat] minimax: Support MiniMax-M3.1-Flash: #45925
- [Perf] moe: Run the Marlin GPT-OSS SwiGLU through the fused Triton kernel: #46134
- [Perf] minimax-sparse: Speed up MiniMax-M3 sparse attention on Hopper: #46135
- [Perf] dsv41: Let Mega-mHC write the MegaMoE input under MegaGate: #46163
- [Perf] dsv41: Read Engram rows from peer device shards: #46180
- [Perf] [NVIDIA] deepseek-v4.1: Default aggregated servers to MegaAttention only when each rank holds 64 query heads (P90 interactivity up to +27% at TP4): #46263
- [Perf] [NVIDIA] kimi-k3: FP32 warp split-K GEMM for the MoE front projection on SM107: #46464
Kernel Library
- [Fix] attn_residual: int64 program ids in the score/combine kernels (int32 overflow at 37,451 tokens): #43438
- [Perf] async-probe: Fuse asynchronous NaN validation into one kernel: #43720
- [Feat] [NVIDIA] megakernels: Add --enable-nvidia-megakernel and a layer for NVIDIA megakernel plugins: #44945
- [Perf] qknorm: run head_dim-128 rows two per warp with 16B lanes, bit-identical: #45214
- [Fix] [NVIDIA] jit: Build sm_100f on CC 10.7 until the bundled CUTLASS supports SM107: #45911
OpenAI-Compatible API
- [Feat] parser: Enrich chat_parsing streaming events: #40478
- [Feat] parser: Add checkpoint-driven response-template adapters: #40479
- [Fix] glm47: Disable tool grammar for
tool_choice="none": #41233 - [Fix] parser: Accept whole-number floats for integer tool arguments: #43941
- [Fix] chat-template: Pass return_dict=False to apply_chat_template(tokenize=True) for transformers 5: #44020
- [Fix] tokenizer_manager: re-report only emitted customized_info keys on cumulative streams: #44537
- [Feat] http_server: Make the multi-worker ASGI app path configurable: #44583
- [Fix] metrics: record HTTP request metrics in tokenizer workers: #44663
- [Fix] muse: Support Responses namespace tools and defer_loading: #44685
- [Fix] Inkling: frame constrained output in native message tokens: #44716
- [Fix] dsv41: Preserve tool-result image order and current-turn thinking: #44751
- [Perf] function_call: stop rescanning unclosed GLM tool calls per chunk: #45143
- [Feat] glm-5.3: Render GLM-5.3 and GLM-5.3-Flash chat prompts with a Python chat template (prompt rendering 4.7x to 17x faster): #45154
- [Feat] entrypoints: add a chat encoding adapter registry: #45300
- [Feat] openai: Render DeepSeek-V4/V4.1 chat prompts with deepseek-recipe: #45315
- [Fix] http_server: Make SageMaker /ping reflect server readiness: #45445
- [Feat] srt: Balance uvicorn tokenizer workers with per-worker SO_REUSEPORT listeners: #45482
- [Fix] http_server: Apply SGLANG_TIMEOUT_KEEP_ALIVE to Granian HTTP/1.1 connections: #45760
- [Feat] openai: parse DeepSeek output through deepseek-recipe: #45907
- [Fix] minimax-m3: Type nested tool-call arguments and count self-opened reasoning: #46133
- [Feat] openai: Parse DeepSeek text for the native endpoints through deepseek-recipe: #46219
Rust Server
- [Feat] rust/grpc: Add ListModels, Detokenize, and Abort to api.v1: #42118
- [Fix] sglang-server: Keep the matched stop token in output_ids: #42637
- [Feat] sgl-router: Serve native gRPC ChatComplete: #42674
- [Feat] sgl-router: Serve the remaining native gRPC inference RPCs: #42681
- [Feat] sgl-router: Separate LoRA block IDs from prompt lookup keys: #42818
- [Feat] sgl-router: Add --dp-rank-policy round_robin for --dp-aware: #43511
- [Feat] sgl-router: Render Kimi-K3 through sglang-processor: #43608
- [sgl-router] Render GLM-5.3 and GLM-5.3-Flash through sglang-processor: #43625
- [Feat] sgl-router: Render Qwen3.8 (Max, 27B, Flash-Next) through sglang-processor: #43689
- [Rust] Serve /weights_checker from the Rust frontend: #43926
- [Rust] Serve scheduler metrics on the Rust frontend's /metrics: #43942
- [Fix] sgl-router: Send disagg_prefill_dp_rank to decode under --dp-aware: #44072
- [Fix] sgl-router: Resend on a closed pooled connection; add --disable-circuit-breaker: #44075
- [Fix] sglang-server: Honour extra_key on the Rust frontend's /generate: #44207
- [Feat] sgl-router: skip engines at an admission cap in power-of-two; forward x-sglext headers: #44218
- [Fix] sglang-processor: Follow Python's OpenAI layer through #43027, #37577, #43006, #43026, #43005: #44243
- [Feat] sgl-router: Fix cache-aware routing for MTP/EAGLE on hybrid (Mamba/KDA) models: #44252
- [Fix] sgl-router: Receive kv-indexer-bridge event batches outside tokio's coop budget: #44305
- [Feat] rust-server: Apply the x-override-* request headers: #44312
- [Feat] rust-server: Attach configured metadata headers to a healthy /health_generate: #44315
- [Feat] sgl-router: Add --decode-dispatch-limit to pick decode DP ranks near KV arrival: #44402
- [Feat] rust-server: Report cached_tokens and reasoning_tokens in meta_info: #44531
- [Feat] sglang-server: serve /v1/loads from the Rust frontend: #44532
- [Fix] rust-server: Honour cache_salt and require_reasoning on /generate: #44533
- [Feat] rust-server: Add HiCache storage management endpoints: #44726
- [Feat] rust-server: Report Python's launch-time /server_info fields from the Rust frontend: #44764
- [Feat] sglang-server: Serve /flush_cache from the Rust frontend: #44766
- [Feat] rust-server: Serve /v1/tokenize and /v1/detokenize from the Rust frontend: #44767
- [Fix] sgl-router: Keep the kv-indexer connection through bursts of cancelled prefix queries: #45100
- [Feat] sgl-router: Add --pd-stream-after-prefill: #45102
- [Feat] sgl-router: Route prefill buckets by uncached tokens: #45103
- [Fix] sgl-router: Refuse prefill workers that cannot compute before bootstrap under --decode-dispatch-limit: #45120
- [Feat] rust-server: Let the launcher set the default max_new_tokens for /generate: #45431
- [Fix] rust-server: Drop the fixed 64-item multimodal cap in favor of --limit-mm-data-per-request: #45432
- [Feat] rust-server: Render, tokenize and serve chat through sglang-processor: #45449
- [Fix] rust-server: Answer health probes a busy scheduler skipped on the response ring: #45475
- [Fix] rust-server: Match Python's chat layer on templates, streaming and lowered-chat errors: #45659
- [Fix] sgl-router: Do not abort forwarded requests on reqwest's default 30 s TCP_USER_TIMEOUT: #45696
- [Feat] rust-server: Serve /abort_request from the Rust frontend: #45888
- [Fix] rust-server: Deliver scheduler-side aborts to Rust frontend clients: #45899
- [Feat] rust-server: Let a multimodal processor finalize item layout after caller hashes: #45914
- [Fix] rust-server: Keep a transfer backend's own bootstrap port on prefill: #46066
- [Fix] rust-server: Start a transfer backend's own KV bootstrap server on prefill: #46074
- [Feat] rust-server: Report weight versions and media token counts in generate meta_info: #46205
- [Feat] rust-server: Let a multimodal processor check and fetch media during admission: #46206
- [Feat] rust: Render DeepSeek-V4/V4.1 prompts with deepseek-recipe in sglang-processor and sgl-router: #46243
- [Feat] sgl-router: Confirmed-first pending prefixes for cold prompts, with a follower cap: #46505
- [Feat] rust_server: Run --sidecar once per node: #46635
- [Feat] grpc: Advertise media inputs in GetServerInfo: #46882
Simulator
- [Feat] simulator: Add baseline collection and ML latency predictor training tools: #44299
SGLang-Diffusion
- [Perf] diffusion: fuse Wan VAE DupUp3D shortcut add: #32860
- [Feat] diffusion: selectable sequence-parallel strategies for LingBot causal attention: #34385
- [Perf] diffusion: Accumulate chunked LoRA merges with addmm (1.7x faster chunked LoRA merges (geometric mean)): #37620
- [Diffusion] Fuse the Wan VAE conv bias epilogue and tile the RMSNorm+SiLU kernel: #38650
- [Feat] diffusion/model: support Cosmos Dreams and Cosmos Multiview: #39475
- [Fix] diffusion/lora: revalidate FLUX.2 FP8/NVFP4 fast paths after LoRA attach: #40171
- [Diffusion] Qwen-Image 2.1 VAE decoder fast paths: bit-exact norm/bias fusions by default, channels_last decode under quality=extra-high: #41340
- [Perf] diffusion: fused QKV norm + RoPE, cached RoPE tables for FLUX 3 video VAE: #41894
- [Perf] flux2: Build the single-block output projection without a cat: #41943
- [Perf] diffusion: fuse FLUX3 SwiGLU with rowwise FP8 quantization: #42758
- [Diffusion] Bound frame_interpolation_exp to RIFE's supported range: #43014
- [Diffusion] comfyui: consolidate 6 unrelated plugin fixes: #43493
- [Fix] comfyui: Fix GGUF detection, import guard and VRAM accounting in SGLDiffusion plugin: #43536
- [Feat] comfyui: video-to-video input and prompt enhancement for server-mode nodes: #43673
- [Feat] comfyui: Apply ControlNet residuals in Flux integrated mode: #43698
- [Diffusion] cache-dit: respect CFG parallel for dual-transformer models: #43704
- [Fix] diffusion: keep ZMQ ingress sockets on their configured port: #43732
- [Fix] wan: fix opt-in Wan2.2 progressive mode and one-step transitions: #43768
- [Feat] VSA-H3: support sm80 (A100) with a raw-pointer Triton load path (FastH3 on A100 in 59.6 s instead of 85.0 s): #43823
- [Diffusion] comfyui: grouped SGLDOptions, native model loader and model-aware preflight: #44058
- [Diffusion] Z-Image: replace padded rows without a host sync: #44069
- [Feat] wan-animate-2: Support the distilled model: #44085
- [Fix] wan-animate-2: Size the in-context slot from both input grids: #44090
- [Perf] diffusion: convrot_int8 comfy_kitchen issues one GEMM per layer by default: #44179
- [Perf] MiniMax-H3: fused indexed modulation for FP32 AdaLN tables: #44180
- [Perf] diffusion: fused gate/up + SwiGLU MLP on comfy_kitchen INT8 for Qwen-Image 2.1: #44196
- [Fix] diffusion: Invalidate merged LoRA cache after adapter rewrites: #44204
- [Fix] flux: use true_cfg_scale as the FLUX CFG weight: #44260
- [Fix] diffusion: Align Qwen-Image 2.1 resolution and handle RGBA JPEG output: #44275
- [Fix] diffusion: Fix VMoBA chunk selection bugs: #44286
- [Feat] diffusion: Add support for the MOSS-Image family: #44325
- [Feat] diffusion: support FLUX 3 Image Klein (peak memory 54.3 to 29.5 GiB with a 2048x2048 reference on GB300): #44418
- [Fix] diffusion: stabilize FLUX.3 serving and reuse static conditioning: #44493
- [Feat] diffusion: Add opt-in pi0.5 completion-boundary action batching: #44634
- [Perf] diffusion: Cache the fused linear-GELU verdict per linear: #44643
- [Feat] diffusion: Per-task prompt enhancer config and the FLUX.3 Klein upsampler protocol: #44662
- [Perf] [NVIDIA] diffusion/FLUX.3: Run dynamic LoRA through PrimsTS GEMMs: #44761
- [Fix] diffusion: Unmerge LoRA layers the next adapter does not cover: #44770
- [Feat] diffusion: Add worker-side ComfyUI compatibility patches: #44841
- [Feat] ComfyUI: Add online INT8 quantization for BF16 Qwen-Image-2.1 single files (about 1.4x faster than BF16 on L40S with half the DiT weight memory): #44865
- [Fix] diffusion: runtime utils bug fixes: #44887
- [Fix] diffusion: Prevent int32 overflow in Qwen-Image-2.1-Turbo VAE RMSNorm: #45118
- [Perf] diffusion/minimax-h3: Batch the video VAE decoder tiles and fuse the output chain: #45179
- [Feat] diffusion/disaggregation: Split FLUX.3 prompt and image encoders: #45309
- [Perf] cosmos3: optimize v2v preprocess performance: #45398
- [Feat] diffusion: Support FastH3 Omni (Ref2VA 8-step PDD): #45530
- [Perf] diffusion/preprocess: Extend exact GPU input processing across pipelines: #45733
- [Feat] diffusion/openai: Report per-step progress for async video jobs: #45919
- [Fix] diffusion/ltx: Keep tiled VAE decode off the sharded convs under parallel decode (E2E 4.7% lower and 1.7 GB less memory per rank at 768x512): #45942
- [Fix] diffusion: Keep dynamic LoRA deltas on fused linear-GELU sites: #45943
- [Perf] diffusion: Shard LTX-2 stage 2 over the CFG ranks as Ulysses: #45965
- [Perf] diffusion: Cut LTX-2 denoising-step host overhead: #45966
- [Perf] diffusion: Remove per-step and per-request host syncs in Cosmos3 and Qwen-Image-2.1: #45967
- [Perf] diffusion: Turn on breakable CUDA graphs by default for LTX-2/LTX-2.3 two-stage serving: #45977
- [Perf] diffusion: Keep every component resident in auto mode when memory allows: #45978
- [Perf] diffusion: Serve image responses from in-memory encoded bytes: #45987
- [Fix] diffusion: Step MOVA latents outside the serial-CFG branch: #45993
- [Fix] diffusion: Give every SP rank the same Wan TI2V timestep: #45994
- [Fix] diffusion: Offset LTX-2.3 BCG video RoPE by the SP shard start: #45995
- [Fix] diffusion: Fail closed after local worker loss or multi-rank OOM: #46025
- [Fix] diffusion/zimage: Stop serving a previous request's RoPE tables: #46044
- [Perf] diffusion: Keep shared denoising paths off the host and add host-sync test guards: #46045
- [Perf] diffusion: Remove per-step host syncs and uploads in image models: #46046
- [Perf] diffusion: Remove per-step host syncs and uploads in video models: #46047
- [Fix] diffusion/server: Clean up rejected and cancelled video requests: #46057
- [Fix] diffusion/api: Publish image inputs atomically: #46060
- [Fix] diffusion/storage: Join uploads before request cleanup: #46068
- [Fix] diffusion/mesh: Guard asynchronous job publication and defaults: #46081
- [Fix] diffusion/loader: Preserve checkpoints on read failures: #46095
- [Perf] diffusion: Write uncompressed PNG responses without Pillow's row filters: #46099
- [Perf] diffusion/qwen-image-2.1: Keep an image prefix's K/V in the packed buffer across steps: #46101
- [Perf] diffusion: Serve FA4 head sizes on Blackwell instead of falling back to sm80 SDPA: #46102
- [Fix] diffusion/disaggregation: Keep decoded images in tensor transport: #46104
- [Perf] diffusion: Project LTX-2 prompt K/V once per denoising loop: #46105
- [Perf] diffusion: Drop LTX-2's all-valid prompt mask under lossless quality: #46106
- [Perf] diffusion: Route dense attention with a sub-tile side to FA4 on Blackwell: #46107
- [Perf] diffusion: Run LTX-2 stage 2 on a premerged resident DiT: #46108
- [Perf] diffusion: Build one LTX-2.3 AdaLN row per sample when every token shares the sigma: #46109
- [Perf] diffusion: Cut LTX-2.3 per-block elementwise work (contiguous AdaLN values, gate-folded masks, vectorized head gate): #46110
- [Perf] diffusion: Vectorize the LTX-2 Q/K RMSNorm + split-RoPE kernel: #46111
- [Fix] diffusion/disaggregation: Isolate malformed request and result messages: #46120
- [Feat] diffusion: add Hy-Nomas offline and realtime support: #46124
- [Fix] diffusion/server: Clean up workers on startup failure: #46142
- [Fix] diffusion/disagg: Recompute automatic topology per role: #46144
- [Perf] diffusion: Prefer FA4 over cuDNN SDPA for long self-attention on B300: #46146
- [Perf] diffusion: Speed up Wan's spatial-parallel VAE decode: #46149
- [Fix] diffusion/openai: Bound job lists and complete cursor pagination: #46156
- [Fix] diffusion/scheduler: Isolate invalid IPC requests: #46157
- [Fix] diffusion/api: Do not publish failed generation outputs: #46162
- [Fix] diffusion/http: Allow bounded graceful shutdown: #46169
- [Perf] diffusion: Encode Cosmos3 video while it decodes: #46172
- [Perf] diffusion: Run Cosmos3's multi-GPU VAE decode on the lossless fast path: #46173
- [Perf] diffusion: Build spatial-parallel causal conv inputs in one pass: #46174
- [Perf] diffusion: Walk several row chunks per warp in the CuTe-DSL norm kernels: #46227
- [Perf] diffusion: Fuse Wan's q/k RMSNorm with its interleaved RoPE: #46228
- [Perf] diffusion: Split LTX tiled VAE decode across the decode-parallel ranks: #46237
- [Perf] diffusion: Validate breakable CUDA graphs for LTX-2.5 two-stage serving: #46239
- [Perf] diffusion/worker: Move cyclic GC out of request execution: #46493
- [Fix] diffusion: Resolve cached named weights without the Hub API: #46679
Local & Desktop AI
- [Feat] paged-experts: Serve MoE models whose weights do not fit VRAM (12x to 30x faster single-stream decode than
--cpu-offload-gbon a 16 GB card): #42557 - [Perf] [NVIDIA] fp8_fa_sm120: Use the block-scaled MMA encoding (1.47x faster at the kernel level): #43390
- [Fix] [SM120] quantization: Requantize block-FP8 MoE weights to UE8M0 for DeepGEMM: #43556
- [Feat] paged-experts: AWQ, compressed-tensors, MXFP4, NVFP4, W8A8 and block-INT8 experts: #45787
- [Fix] mlx: Accept draft_parallel_layout in the stub's alloc_memory_pool: #46441
AMD / ROCm
- [Perf] [AMD] attention: Enable diffusion BF16 attention and MXFP4 compilation: #27702
- [Feat] [AMD] disagg: Support DSV4 decode radix cache on ROCm unified KV (bf16/fp8, DSpark) (long-context KV transfer 68% to 72% smaller): #30929
- [Fix] [AMD] custom-allreduce: Stop registering oversized inputs in the deterministic all-reduce: #34543
- [Fix] [AMD] dsa: Fix AITER prefill with an FP8 KV cache: #39083
- [Perf] [AMD] quark: Add opt-in AITER ASM FP4 GEMM for Quark dense linear: #39598
- [Perf] [AMD] dsv4: fp8 target lm_head GEMM for DSPARK verify logits: #39622
- [Fix] [AMD] Quark: fix Quark block-FP8 loading in DeepSeek loader: #40776
- [Perf] [AMD] attention: Spread uneven Triton decode batches across XCDs: #41229
- [Fix] [AMD] mamba: Keep a chunked prefill cut on the mamba chunk grid so DCP can donate checkpoints: #41701
- [Perf] [AMD] quark/Qwen3.8: Quantize the BF16 shared expert of Quark MXFP4 checkpoints to MXFP4 while loading for shared-expert fusion (+4.5% output throughput): #41899
- [Perf] [AMD] dsv4: reuse the q_lora fp8 quant for the C4 indexer wq_b (4/8): #41903
- [AMD][Diffusion] Route unquantized linears through aiter tgemm on gfx1250 (opt-in with
SGLANG_USE_ROCM_AITER_TGEMM=1; Wan2.2 denoising 3.55x faster on MI450): #41988 - [Perf] [AMD] qwen3.5: Fuse activation quant into small-M FP8 block-scale GEMM on gfx950 (Qwen3.5 interactivity 5% to 10% higher at TP8): #42229
- [Fix] [AMD] hicache: Serve encoder-replay prefix hits with a paged DSpark draft SWA: #42716
- [Perf] speculative: Parallelize EAGLE spec-decode KV index building over token blocks (flashinfer draft decode; verify/draft-extend for flashinfer, flashinfer_mla, aiter): #42803
- [Fix] [AMD] aiter: Fix garbage decode output for non-sliding-window models on the vectorized_5d KV layout: #43335
- [AMD] Use the preallocated EAGLE topk=1 draft chain on HIP: #43640
- [Perf] [AMD] dsa: Run DSA prefill under breakable CUDA graphs on HIP: #43642
- [Perf] [AMD] dsa: Enable the gfx950 fused DSA indexer for block-FP8 wq_b/wk (GLM-5.3 MXFP4 P50 interactivity +6.8%): #43643
- [Fix] [AMD] diffusion: Compare ReduceOp by identity so the compiled TP all-reduce stays one graph: #43770
- [Fix] [AMD] deterministic: Refuse RCCL all-reduce in deterministic mode above TP2: #43809
- [Fix] [AMD] diffusion: Prevent ComponentResidencyError after compile warmup: #43952
- [Fix] [AMD] kda: Match the fused verify Q/K norm to its reference kernel: #43991
- [Perf] [AMD] dsv4: fp8 verify writes the SWA ring in the fused norm/rope kernel (2/8): #44003
- [Perf] [AMD] moe: small-batch sort with histogram + bitonic sort for 32 < P <= 256 (3/8) (decode step about 4% lower): #44004
- [Perf] [AMD] dsv4: FP4 indexer on aiter's row-group kernel for decode and target-verify (5/8): #44006
- [Perf] [AMD] dsv4: optional e8m0 mxscale GEMM for dense 128x128 fp8 linears (6/8): #44007
- [Perf] [AMD] dsv4: fp8 unified_kv decode/verify on FlyDSL v4 nm decode with a latency-model split plan (1/8) (TP8 decode step 3.1% to 3.9% lower): #44016
- [Fix] [AMD] activation: Run ReLU2 natively on ROCm: #44198
- [Fix] [AMD] qwen3: Fix KV cache write in the fused QK-norm + mRoPE decode path (Qwen3-VL GSM8K 0.005 to 0.942): #44266
- [Perf] [AMD] qwen3: Fuse QK-norm + RoPE + KV cache write for dense Qwen3 decode (TPOT up to 10.5% lower): #44267
- [Feat] [AMD] monokernels: one runtime switch for vendor monokernel plugins (DeepSeek-V4.1-Flash single-request decode +75% with DSpark): #44353
- [Perf] [AMD] dspark: No per-step host sync in decode (1/4) (DeepSeek-V4.1-Flash DSpark TPOT 6.8% lower at concurrency 1): #44405
- [Fix] [AMD] mori: Look up elided DSA state entries by the paired dst index: #44412
- [Perf] [AMD] monokernels: Let a plugin skip per-step attention metadata: #44601
- [Fix] monokernels: Honor resolved CUDA graph configuration: #44630
- [Perf] [AMD] dsv4: Add a separate MegaMoE decode instance: #44657
- [Fix] [AMD] diffusion: Keep gfx1250 FlyDSL BF16 attention out of torch.compile: #44671
- [Fix] [AMD] diffusion: Accept residual and skip_bias in the ROCm Conv3d-to-Conv2d VAE patch: #44672
- [Perf] [AMD] deepseek-v4: Skip top-k refinement for proven positive-zero ties (P90 interactivity +8.2%): #44927
- [Fix] [AMD] deepseek-v4.1: Reject attention DP with encoder SWA bounded replay on ROCm: #44934
- [Perf] [AMD] deterministic: Faster extend attention launch on gfx950 (deterministic-mode prefill 21% to 130% faster on long prompts): #45098
- [Perf] [AMD] attention: Restore the gfx950 extend tile for head_dim <= 128 (+28% throughput at 32K input with the Triton backend): #45099
- [Perf] [AMD] dsv41: run gfx950 MXFP8 dense GEMMs on aiter's preshuffled FlyDSL kernel (DeepSeek-V4.1-Flash decode step up to 4.8% lower on MI355X): #45123
- [Perf] [AMD] dsv41: faster mHC boundary Triton tile for 96+ rows, bitwise unchanged: #45124
- [Fix] [AMD] grok: accept allow_patterns_overrides in presharded _prepare_weights patch: #45217
- [Fix] [AMD] diffusion: Enable
--quantization fp8on gfx1250 (torch.compile lowering + per-channel weight scales): #45304 - [Perf] [AMD] dsv41: Skip unneeded layer 20 computation under decoder-bounded-replay: #45339
- [Fix] [AMD] gpt-oss: support EP for gfx1250 W4A8 MXFP4 MoE: #45365
- [Fix] [AMD] dsv41: Set moe act to fp8 by default: #45480
- [Fix] [AMD] attention: Pass the layer softcap to aiter unified-attention decode: #45531
- [Perf] [AMD] kda: Use a HIP chunked-prefill kernel for KDA on gfx950 (Kimi-K3 single-request TTFT 11.9% lower at 1K input): #45710
- [Perf] [AMD] kda: Use a HIP fused decode kernel by default for KDA (Kimi-K3) on gfx950: #45713
- [Perf] [AMD] Qwen3-Next: fused QK Gemma RMSNorm + partial RoPE on gfx950: #45739
- [Perf] [AMD] GPT-OSS: Pick the gfx1250 W4A8 MoE backend, in-kernel GEMM 1 gather (opt-in with
SGLANG_GFX1250_W4A8_KERNEL_GATHER=1; gpt-oss-120b W4A8 output throughput +30%, TTFT 40% lower): #45758 - [Fix] [AMD] diffusion: Fix the first-step denoise regression from compiling ENABLE_CK=0 BF16 attention: #45838
- [Fix] [AMD] rotary: Pass a query-dtype cos_sin_cache to the HIP fallback: #46161
- [Fix] [AMD] MiniMax-M3: default SGLANG_MINIMAX_M3_INDEX_TOPK_FREQ to 1: #46340
- [Perf] [AMD] diffusion: Route large BF16 linears to the FlyDSL a16w16 GEMM on gfx1250 (FLUX.2-dev 1.21x to 1.28x faster end to end on MI450): #46366
- [Fix] [AMD] aiter: Fix garbage output with an fp8 KV cache and calibrated KV scales: #46528
NPU / Ascend
- [NPU] [Fix] Fix rope sin cos pre-calculation for Qwen3.5 during Pipeline Parallelism: #29753
- [Fix] [NPU] qwen3.5: Fix dense serving on Ascend 950: #32745
- [Feat] [NPU] llguidance: add bitmask path for Ascend: #36074
- [Perf] [NPU] attn: add not needs_cpu_seq_lens for dsa modules: #38461
- [Fix] [NPU] scope: fix dtype error in sampling backend ascend: #39609
- [Fix] [NPU] attention: Use FIA TND for MLA prefix attention and add Gemma norm fallback: #40131
- [Fix] [NPU] glm5.2: support pcp v2 with interleave & zigzag: #40165
- [Fix] [NPU] dspark: Fix attn_tp > 1 draft wo_b all_reduce group: #40399
- [Fix] [NPU] requantization: gate arch35 block-FP8 -> MXFP8 requantization: #40435
- [Fix] [NPU] qwen: Fix shared-expert routing with Ascend FuseEP: #40516
- [Perf] [NPU] deepep: enable MXFP8 low-latency dispatch for Fp8MoEMethod FP4 experts (MiMo-V2.5-Pro with DFlash mean TTFT 21.8% lower): #40519
- [Perf] [NPU] mem_cache: Add optional ScatterPa KV cache writes: #40535
- [Fix] [NPU] qwen: adapt pcp-v2: #41564
- [Fix] [NPU] glm5.2-mtp: L2 cache adaptation: #41642
- [Fix] [NPU] mla-pool: DCP allocator page and index-K bound, short-pool warning: #42976
- [Fix] [NPU] pre_warm_nccl: init npu backend before the parallel runtime touches a device: #43039
- [Perf] [NPU] glm5.2: Support dynamic INT8 MLAProlog for attention (decode latency 8.4% lower): #43469
- [Fix] [NPU] disaggregation: Fix host-pool backup for decode retraction: #43814
- [Fix] [NPU] kimi-k3: skip fused gmm1 for situ activation in Ascend runner: #43837
- [NPU][Fix] use CANN libdevice for DSpark Markov sampling: #43849
- [Perf] [NPU] mem_cache: Increase GLM-5.2-W4A8C8 KV cache capacity with compact DSA indexer allocation: #44436
- [Perf] [NPU] scheduler: Remove blocking seq_lens D2H from graph replay: #45170
- [Perf] [NPU] AscendAttnBackend: D2H-free block-table gather for mirrorless DSA graph replay: #45949
CPU / Intel / XPU
- [Feat] [Intel] hicache: Enable
L1<->L2support on XPU: #32503 - DSPARK enable for XPU: #33739
- [CPU]: Add group-aware CPU SHM collective kernels: #35435
- [Feat] [CPU] model: Support LTX and MOVA models on CPU: #36632
- [Perf] [XPU] usp: Enable packed Ulysses QKV all-to-all: #39989
- [Perf] [XPU] fp8: Optimize FP8 scale layouts: #40052
- [Fix] [CPU] memory: Honor cgroup memory.max when sizing CPU memory: #40066
- [Feat] [Intel] disaggregation: Support XPU for staging-buffer KV transfer: #40861
- [Fix] [XPU] runtime: Fix XPU rotary embeddings and tokenizer padding: #41213
- [Perf] [XPU] diffusion: Use the AOT SYCL fused residual-norm-scale-shift kernel: #41401
- [Perf] [XPU] flux2: Fuse bit-exact LayerNorm and gated residual norm: #41942
- [Perf] [XPU] diffusion: Avoid QKV copies in flash attention: #41944
- [XPU] Use the shared fused decode-metadata kernel in the intel_xpu ba…: #42122
- [Feat] [CPU] Grok-model: enable grok-2 model on xeon cpu: #43341
- [Perf] [XPU] server_args: Enable fused QK-norm + RoPE by default: #43345
- [Feat] platform: Add initial T-HEAD SAIL platform support [2/N]: #43826
- [Fix] [XPU] gemma3: Enable bidirectional image attention: #44225
- [Feat] [CPU] model: Support Muse models on CPU: #44624
- [Feat] [CPU] model: Support Cosmos3 series models on CPU: #45478
- [Fix] [CPU] memory: budget overestimation caused by filtering empty NUMA nodes: #45516
Security
- [Fix] model_loader: Avoid unsafe pickle fallback when loading PyTorch checkpoints: #42760
- [Fix] diffusion: OpenAI API hardening: #43585
- [Fix] utils: Allowlist torch.bcomplex32 in SafeUnpickler: #44440
- [Fix] sgl-router: Bump rustls to 0.23.46 for RUSTSEC-2026-0346..0349: #44509
- [Fix] srt: Admit only allowlisted peers on TCP-bound ZMQ IPC sockets: #45250
- [Fix] srt: Pin the source address of TCP ZMQ connects for the peer allowlist: #45437
- [Fix] sgl-router: Bump rustls-webpki and webpki-roots for emSign advisories: #45526
- [Fix] sgl-router: Bump tokio for RUSTSEC-2026-0386: #46260
Cookbook Updates
- DeepSeek-V4 and DeepSeek-V4.1: a DeepSeek-V4.1-Flash 1017 variant, PD with DSpark, refreshed AMD recipes for V4 and V4.1 (including the PD image and c1 capacity fix), and Ascend A3 / 950 single-node recipes: #44937, #44848, #44368, #44454, #44903, #45070, #45297, #45693, #45602
- SemiAnalysis InferenceX AgentX: launches for Muse Spark Open, GLM-5.3 and MiniMax-M3.1-Flash, and refreshed points for GLM-5.2, Qwen3.5, Qwen3.8-Flash-Next and DeepSeek-V4.1-Flash on B300, GB200 and GB300: #44065, #44228, #44654, #44712, #45039, #45111, #45311, #45495, #45513, #45749, #46005, #46506, #46547
- Muse Spark Open: a new page with GB300 Dynamo deployments, an MI355X AgentX recipe and tuned GB300 BF16 recipes: #44527, #45629, #45689, #45422, #45429
- MiniMax-M3.1: a new page with GB200 recipes and a B200 low-latency monokernel recipe: #45924, #46546
- Beam 1.0: a new page with Python and Docker commands: #44694, #45991
- MiniCPM-V 4.7: DSpark benchmarks refreshed at the final model PR: #44310
- FLUX.3 Image Klein: an expanded model description and a prompt upsampler section: #44515, #44512
- Cosmos3 and diffusion guides: prompt enhancer tasks, Cosmos recipe links, reference resizing, LoRA auto mode and image profiling: #45705, #44377
- GLM-5.2 and GLM-5.3: Ascend deployment recipes for GLM-5.2, GLM-5.3 and GLM-5.3-Flash, and a TPU7x recipe for GLM-5.3-Flash: #43372, #45597, #45642, #45326
- Qwen3.5, Qwen3.6 and Qwen3.8: Qwen3.5 MI355X FP8 agentic tuning with HiCache write-through, an RTX Spark Qwen3.6 NVFP4 recipe, Ascend A3 scenarios for Qwen3.6, FP8 KV and the QSA indexer by default for Qwen3.8-Flash-Next, and the released Qwen3.8-Max W4A8 checkpoint on Ascend: #46017, #46032, #44707, #38678, #45245, #45574
- Jais 3: a new page (runs on the K2 Horizon architecture): #42766
- Ornith-1.5: a new page (runs on the Qwen3.5 architectures): #45455
- New pages for models that run on preview images in this release: Ling-3.1-flash (#44301, #44307), Nemotron 3.5 Super VL (#44510, #44587), Step 5 Preview (#44357) and Recursive Flint: #45625
- TPU: the README and TPU guide introduce SGLang-TorchTPU next to SGL-JAX: #45246
- Ascend NPU docs move to CANN 9.1.0: #44177, #44432, #44695
Dependencies
- FlashInfer 0.7.1 to 0.7.2: #44591
- sglang-kernel 0.4.9 to 0.4.10 (#43922), sgl-deep-gemm 0.2.2 to 0.2.3 (#44666) and then 0.2.3.post2 (#45933), and the new sgl-flash-mla 1.0.0: #45893
- New dependencies: deepseek-recipe 0.2.0 for DeepSeek-V4 / V4.1 chat (#45315) and tml-renderers 0.1.0 for Inkling chat: #45340
tokenspeed_mlais no longer installed (#44803) andcuda-tileis dropped: #44393- OpenAI SDK 2.6.1 to 2.8.0 (#45291), uvicorn >= 0.51 (#45482), timm 1.0.16 to 1.0.25 (#46620), python-dateutil < 3 to keep botocore resolvable (#45857), and sgl-eval 0.1.3: #44131
- Docker: CUDA images ship the sgl-nvidia-mega megakernel plugin, inert unless
--enable-nvidia-megakernelis set (#44945, #46030); AWS EFA image variants (#41006) andnumactlin runtime images forSGLANG_NUMA_BIND_V2: #43883 - ROCm: a ROCm 10.1 image for gfx1250 alongside ROCm 10.0 (#42699), the sgl-amd-mono monokernel plugin in the ROCm image (#44621, #45389), AITER pins
2bb549102(#44379) and285579b62for gfx942 / gfx950 (#45498), and Mooncake v0.3.14-rc1 for cross-node KV transfer: #44404 - XPU: oneAPI 2026.1.3 and sglang-kernel-xpu 0.4.0 to 0.5.0: #43989
- MLX: mlx >= 0.32.4: #44367
- T-HEAD SAIL: a new
srt_sailextra (inpyproject_other.toml) for the initial platform support: #43826
Breaking Changes & Upgrade Notes
- DeepSeek-V4 and V4.1 chat now follows deepseek-recipe. Prompts, outputs and the native
/separate_reasoningand/parse_function_callendpoints go through DeepSeek's official encoder, and SGLang's ownencoding_dsv4.py/encoding_dsv41.pyare gone. Clients may notice: streamed tool-call arguments arrive in pieces (the first chunk carriesarguments: ""), so concatenate them as the OpenAI SDK does; tool-call ids use the officialcall_00_...format; V4 reasoning effort follows DeepSeek's official profile and thedsv4_reasoning_effort_profileoverride is removed; a numericreasoning_efforton V4 returns 400;developermessages render as system messages;/parse_function_callalso returns calls to tools outsidetoolswithtool_index-1; requests that combinetoolswithcontinue_final_messagereturn 400. Output parsing goes through deepseek-recipe when the DeepSeek reasoning or tool-call parser is set; sgl-router and sglang-server keep their Rust parsers for now: #45315, #45907, #46219 - DeepSeek-V4.1 decoder SWA bounded replay is on by default for supported CUDA configurations, including both PD roles. It is not numerically identical to full prefill. Pass
--no-enable-decoder-swa-bounded-replayto keep full prefill, for example for RL rollouts whose logprobs must match the trainer: #44403, #45035 - DeepSeek-V4.1 defaults to MegaAttention under PD. On SM100 / SM103 both PD roles now resolve
flashmla_megaand the PD handshake checks the KV layout, so prefill and decode must run the same attention backend (#45075). Upgrade prefill and decode together, since an older decode cannot read the new handshake field; pass--dsv4-attn-backend flashmlaon both roles to keep FlashMLA. Aggregated servers keep FlashMLA when attention is tensor-parallel across ranks: #46263 - The
tokenspeed_mlaattention backend is removed.tokenspeed_mlastill starts as a deprecated alias forcutedsl_mlawith a warning. To switch, pass--attention-backend cutedsl_mla --kv-cache-dtype fp8_e4m3, plus--speculative-attention-mode decodewith speculative decoding: #44803 --speculative-skip-dp-mlp-syncis removed. Launch commands that pass it now fail with an unrecognized-argument error; drop the flag: #45043- The
kitchen_int8diffusion quantization name is removed. Useconvrot_int8, deprecated in favor of it since v0.5.22: #45706 SGLANG_ENABLE_UNIFIED_RADIX_TREEis now ignored without a warning. It already had no effect because the unified radix tree is the default; drop it from launch scripts: #45090- Legacy tar-format
.ptcheckpoints no longer load. The unsafeweights_only=Falsepickle fallback is removed; convert such checkpoints to safetensors or zip-format.ptoffline: #42760 - Diffusion HTTP endpoints validate more strictly. URL values for
upscaling_model_pathandframe_interpolation_model_pathreturn 400, an unsupportedresponse_formatis rejected before generation, and a failed cloud upload returns 502 instead of 400: #43585 - The first startup of a FlashInfer CUTLASS MoE server takes longer. Startup autotune now also covers prefill sizes, so a server with no saved tuning results takes longer to become ready (on DeepSeek-V4.1-Flash on 8x H200, autotune goes from about 41 s to 177 s). Later startups reuse the saved results, and
--disable-flashinfer-autotuneskips tuning: #43363 - Diffusion PNG responses are larger. PNG output now defaults to zlib level 0: the same pixels, about 3x the bytes, several times faster to encode. Set
output_compressionto a value other than 75 to pick a level: #45987 - sgl-router moved to the top-level
sgl-router/directory. Builds and scripts that point atexperimental/sgl-router/must update the path (#45414);--max-in-flightis now--max-inflight-requests, and the old name still works as an alias: #44950
New Contributors
- @jinwukong made their first contribution in #43067
- @xinchen9 made their first contribution in https://github.com/sgl-project/sglang/pull/42122
- @razorback16 made their first contribution in #35757
- @lijrjyan made their first contribution in #31892
- @raghulchandramouli made their first contribution in #43323
- @adityasingh2400 made their first contribution in #33715
- @GeorgeSherif made their first contribution in https://github.com/sgl-project/sglang/pull/42766
- @TwelveNights made their first contribution in https://github.com/sgl-project/sglang/pull/42148
- @Robin930 made their first contribution in #44453
- @vineelbhat made their first contribution in #41303
- @koladefaj made their first contribution in #44213
- @AaravMehta-07 made their first contribution in #43832
- @feixiangpeng made their first contribution in https://github.com/sgl-project/sglang/pull/44790
- @Woody-Hu made their first contribution in #43823
- @Shan2L made their first contribution in #43732
- @heiculi98 made their first contribution in #40171
- @kchatr made their first contribution in #42833
- @Linda-Stadter made their first contribution in #44761
- @jumanzii made their first contribution in #42702
- @0z5a made their first contribution in #39592
- @jiuqiant made their first contribution in #43438
- @hanahhh made their first contribution in #40899
- @Guozhong23 made their first contribution in https://github.com/sgl-project/sglang/pull/40535
- @toulzx made their first contribution in #41233
- @wutaosang made their first contribution in https://github.com/sgl-project/sglang/pull/43981
- @lpdink made their first contribution in https://github.com/sgl-project/sglang/pull/44030
- @XXXXRT666 made their first contribution in https://github.com/sgl-project/sglang/pull/44367
- @sihancx made their first contribution in #39580
- @ZhouYuxuanYX made their first contribution in #40762
- @adi-suresh01 made their first contribution in #44480
- @zydo made their first contribution in https://github.com/sgl-project/sglang/pull/43666
- @Bob-Chen222 made their first contribution in https://github.com/sgl-project/sglang/pull/45326
- @bzf0654-coder made their first contribution in https://github.com/sgl-project/sglang/pull/38678
- @chaokunyang made their first contribution in https://github.com/sgl-project/sglang/pull/45606
- @slyu0021 made their first contribution in https://github.com/sgl-project/sglang/pull/40120
- @JiaShuai0209 made their first contribution in #40424
- @ItsOdeLeo made their first contribution in https://github.com/sgl-project/sglang/pull/45419
- @chaoyujiang made their first contribution in #40900
- @nvrohanv made their first contribution in https://github.com/sgl-project/sglang/pull/45521
- @viraatc made their first contribution in https://github.com/sgl-project/sglang/pull/45629
- @0130w made their first contribution in #44726
- @qidaye made their first contribution in #40565
- @Waterpine made their first contribution in https://github.com/sgl-project/sglang/pull/45422
- @Levius-Fubuki made their first contribution in #37620
- @JohnZhao01 made their first contribution in #45407
- @tatavishnurao made their first contribution in https://github.com/sgl-project/sglang/pull/46078
- @maobaolong made their first contribution in #40595
- @tavorlavi made their first contribution in #45919
- @wl55721 made their first contribution in https://github.com/sgl-project/sglang/pull/39772
- @LongCatchen made their first contribution in https://github.com/sgl-project/sglang/pull/36074
- @liuyd-dev made their first contribution in https://github.com/sgl-project/sglang/pull/45642
- @Monstertail made their first contribution in https://github.com/sgl-project/sglang/pull/45710
- @YangChen-pro made their first contribution in #45143
- @rayanht made their first contribution in https://github.com/sgl-project/sglang/pull/46640
- @Oseltamivir made their first contribution in https://github.com/sgl-project/sglang/pull/46340
- @shijie-lyu made their first contribution in #45445
- @ischencheng made their first contribution in #41275
- @nguyendvk made their first contribution in https://github.com/sgl-project/sglang/pull/42760
Full Changelog: v0.5.22...v0.5.23
Highlights
950 PRs from 245 contributors.
New models
| Model | Type | Cookbook |
|---|---|---|
| Clef / Clef Flash | LLM / VLM | link |
| PPLX-Decider v1 / v1.1 | LLM / VLM | link |
| EmbeddingGemma 2 | LLM / VLM | link |
| LLaDA2.2 | LLM / VLM | |
| HunyuanImage-3.0-Instruct | Diffusion | link |
| Kandinsky 6 / Kandinsky 6 SR | Diffusion | link |
| Wan-Animate-2 | Diffusion | link |
| SANA-Video 2.0 | Diffusion | link |
| LLaDA-Image / LLaDA-Image-Turbo | Diffusion |
What's new
- DeepSeek-V4.1 gets up to 2.3x more throughput per GPU on H200, and now runs on AMD MI350X / MI355X with up to 1.9x throughput from DSpark (#41251, #41308).
- Qwen3.8-Flash-Next decodes 12% faster on B200, and GLM-5.3-Flash with DFlash 10% faster on B300 (#42175, #42489).
- MiMo-V2.6-Pro gets 1.85x decode throughput with DFlash on B300 (#41886).
- Models load 10x faster with the new InstantTensor loader (#40453).
- PD disaggregation gets up to 27% more throughput and half the p95 TTFT on long-prompt workloads (#40703).
- HiSparse now supports EAGLE and MTP for long-context serving with the KV cache in CPU memory (#37771).
- GLM-5.2 and GLM-5.3 can now combine decode context parallelism with MTP, with up to 2.1x faster decoding on GLM-5.3 (#43735).
- SGLang now runs on Apple Silicon Macs with the standard PyTorch backend, alongside MLX (#36780).
- MiniMax-H3 and Qwen-Image 2.1 now run in ComfyUI with SGLang Diffusion, which also loads community GGUF checkpoints and LoRAs (#43504, #43581, #42556, #39523, #42174).
- Important fix: Hybrid linear models could resume from wrong mamba states when speculative decoding and structured output were used together. Thanks to @Wirick and Baseten for finding it (#43029).
To upgrade
SGLang now requires Python 3.11 or newer.
uv pip install --prerelease=allow sglang==0.5.22
| Platform | Docker |
|---|---|
| NVIDIA (CUDA 13) | docker pull lmsysorg/sglang:v0.5.22 |
| AMD MI35x | docker pull lmsysorg/sglang:v0.5.22-rocm10-mi35x |
| AMD MI30x | docker pull lmsysorg/sglang:v0.5.22-rocm10-mi30x |
| Intel GPU | docker pull lmsysorg/sglang:v0.5.22-xpu |
| Intel CPU | docker pull lmsysorg/sglang:v0.5.22-xeon |
Full Release Notes
Speculative Decoding
- [RL][Spec] Introduce top-p mask capture for spec and add DFlash/DSpark impl: #34201
- [HiSparse] Add speculative decoding support with multi-step KV cache swapping: #37771
- [Kimi-K3][DCP][DSpark] fix eager mode nonetype crash due to no extend_prefix_lens: #38991
- [Spec][PP] Fix hybrid recurrent-state commit and micro-batch pairing under PP x speculative decoding: #40001
- [Spec] Reuse K3 auxiliary outputs across decode CUDA graph sizes: #40159
- [NVIDIA][MegaMoE] Fix the default spec decoding moe backend when the target model uses flashinfer megamoe: #40772
- fix(spec): enable Qwen3.5 EAGLE3 capture and streaming overlap: #41154
- [Qwen3.8-Flash-Next] Fuse QSA graph replay metadata across draft steps: #41173
- [Qwen3.8-Flash-Next] Fuse NEXTN verify and draft graph input preparation: #41175
- [HiSparse] fix: reject speculative decoding and mixed chunk at startup: #41347
- The sgalng api specs: #41585
- [Spec] Dspark support: #41623
- Fix XQA draft extend with FP8 KV cache in trtllm_mha: #41656
- [QSA] Enable breakable prefill CUDA graphs for text-only Qwen3.8 Flash-Next (capture-safe metadata, MTP side-channel padding): #41729
- [DFlash] Keep grouped-conv taps inside each request's block so NaN/Inf cannot leak across requests: #41736
- [DFlash2] Keep the greedy selector walk in range for all-NaN score rows: #41737
- [PD][Spec] Give a PD decode's first EAGLE draft step a proposal distribution: #41783
- [Fix] Pass the draft's attention ownership to DFLASH's eager LiLiCorr scope: #41808
- [Spec] Fix Qwen3.5 text model EAGLE3/DFLASH aux-layer capture: #42002
- [Spec][MegaMoE] Let the speculative draft choose its own W4A4 MXFP4 MegaMoE MMA type: #42022
- [Spec] LiLiCorr: named head MLPs and quantized head linears: #42057
- [Spec] Add opt-in block verification: #42297
- [Fix] Load a draft's tensor weight update from its deployment TP rank: #42346
- [Fix] Skip a draft's decode recapture when it owns no decode graph: #42347
- [Perf] Improve GLM-5.3-Flash DFlash decoding on B300 (decode 1091.9 to 1199.7 tok/s on 8x B300 at concurrency 1): #42489
- [DSpark] Mask uncommitted slots before commit-inject cache lookups: #42624
- Back the DFlash-family draft KV pool to the post-capture token count: #42651
- perf(kda): rebalance MTP verify precompute and state synchronization (verify kernel 13.2 to 9.7 µs on Vera Rubin): #42683
- [Spec] Fix wrong tokens from compact-verify CUDA graphs when a token tier is captured more than once: #42709
- [KDA] Support Helion MTP verification and harden prefill, decode: #42857
- [KDA] Default GLM-5-Next spec verify to the CuTe DSL KDA kernel: #42905
- [DSpark] Fold draft sampling for DeepSeek-V4.1 heads under AUTO: #42987
- [Spec] Support block verification for the DFlash family: #43012
- [Scheduler] Use committed length in spec mamba track boundary check (found by @Wirick at Baseten): #43029
- [Spec] Allow Qwen3.5 backends to opt into packed target verification: #43037
- [DFlash] Build the attention QKV projection through an overridable hook: #43179
- [DSpark] Run every markov head through one step interface: #43620
- [DSpark] Add
MarkovSampleStep, one op for a markov draft step's add and token pick: #43621 - [DSpark] Draft the eager markov block through
MarkovSampleStep: #43622 - [DSpark] Run the same markov block in the draft graph tail; drop
SGLANG_DSPARK_FOLDED_SAMPLING: #43668 - [DSpark] Merge the draft TP sync sites into
dspark-draft: #43669 - [DSpark] Add
SGLANG_DSPARK_USE_GREEDY_DRAFT, an argmax draft accepted as one-hot: #43670 - [DeepSeek-V4.1] Admit the NIXL transfer backend for DSpark PD: #43734
- [DSpark] Share draft params, logits dtype, confidence and the accept tail between graph and eager: #43790
Breakable CUDA Graph
- [Fix] Llama4 local attention: read page ids from the graph tables in CUDA-graph capture/replay (page_size > 1): #38187
- [BCG][DeepSeek-V4-Flash][CP] Support DeepSeek V4 prefill with interleave CP and breakable CUDA graphs: #38603
- fix(cuda-graph): remove unnecessary CP batch alignment: #40543
- [GLM-5.3-Flash] Capture the KDA extend in breakable prefill CUDA graphs: #41105
- Drop the GLM-5.3-Flash breakable prefill chunk-size pin: #41837
- [Fix] MiniMax-M3: default to the breakable prefill CUDA graph: #41845
- Size the VMM graph-input exchange by the widest input across ranks: #42125
- [DSA] Fix pooled page table refresh for target verify CUDA graph replay: #42750
- [GLM-5.3-Flash] Enable breakable prefill CUDA graph by default: #42845
- Clamp default decode CUDA graph max_bs to max_running_requests: #43172
- [Fix] Unblock CUDA graph startup for hybrid parallelism: #43211
- Allow mixed chunk prefill on the Full prefill CUDA graph backend: #43461
- [DeepSeek-V4.1] Run decoder SWA bounded replay inside the breakable prefill CUDA graph: #43736
- Fix autograd retention in CUDA graph capture tail hooks: #43854
Attention Backends
- [SM120&90] Add CUDA fused Triton sparse-MLA prefill backend for DSA: #32779
- GLM5.2 DSA Indexer Fallback: #32947
- fix(fa4): isolate sheared-bias workspaces by CUDA stream: #33000
- fix: default Inkling to fa4 on Hopper: #35292
- Fix Kimi-K3 MLA output gate dispatch on non-CUDA devices: #36406
- Prebuild fixup_zero_kv JIT module at TRTLLM-MLA backend init: #37525
- Skip fully-masked KV tiles in sliding-window extend attention (up to 39% faster kernel at 8K tokens): #39769
- [Linear Attention] Expose GDN/KDA prefill hooks and auxiliary cache accounting: #40227
- Support kv cache sharding for DSA indexer: #40925
- [Qwen3.8-Flash-Next] Fuse QSA KV preparation and sparse block expansion: #40972
- [Kernel] Expose FlashMLA kv_format in sgl-kernel sparse decode: #41336
- [DeepSeek-V4][DSA] Name the FlashMLA KV format and drop the V4.1 support probe: #41337
- feat(kda): integrate FlashInfer prefill checkpoints: #41400
- [KDA] Enable the ptx_kda prefill backend on SM100 (B200 / GB200): #41445
- [DeepSeek-V4.1] Add opt-in TRT-LLM sparse attention support: #41603
- [DeepSeek-V4.1] Prefill indexer: raw selection, row-pair schedule ids, no host syncs: #41659
- [Perf] Use FA4 by default for MiMo on SM100 (1.85x DFlash decode throughput on 8x B300): #41886
- [Qwen3.8-Flash-Next] Fuse QSA prefill KV gathering and dtype conversion: #41933
- [DSA] Stop forcing the per-step CPU seq_lens sync for the k-pool indexer: #41987
- [Gemma4][Attention] Enable Gemma4 FA4 on SM90 with head-dim 512 support: #42019
- [Perf] Optimize MiMo local audio attention on Hopper: #42101
- [DeepSeek-V4] Guard V4.1 indexer SM100 check on non-CUDA devices: #42114
- [DeepSeek-V4.1] Bounded replay with sparse mla path: #42273
- fix(dsa): align FA3 metadata with unpadded queries: #42793
- [Fix] Run attention on a padded extend batch's real rows: #42896
- [FA4] Keep stream last in the SM100 MLA kernel so disk-cached objects load: #42926
- [Perf] Route BF16 mixed-batch decode tails through decode attention: #43238
- [KDA] Default safe-gate Blackwell prefill to FlashInfer: #43246
- [QSA] Bound the decode indexer's work by the context length (bs64 decode step 28.75 to 26.54 ms for Qwen3.8-Flash-Next on 4x B200): #43463
- [Kernel] Expose bundled FlashMLA MegaAttention kernels and APIs: #43918
MoE & Expert Parallelism
- [Model] Add LLaDA2.2 Block Routing MoE support: #31768
- DeepEP v2: expanded (do_expand=True) prefill dispatch: #37261
- Add per-token NVFP4 MoE support for ReLU2 activation: #39199
- fuse shared experts with routed experts in MegaMoE's DeepGEMM: #39313
- [MegaMoE] Preserve W13 layout for ModelOpt NVFP4 experts: #39388
- [MoE] Use FlashInfer A2A for prefill instead of AG+RS: #39818
- [Fix] Don't let a failed deep_ep import-time check kill servers that never use DeepEP: #40671
- [Fix] MoE: require TopK layer_id to ensure routed expert captures: #40980
- [Fix] Skip post-experts EP all-reduce on the FlashInfer cutlass FP4 all-gather MoE path: #41072
- [Fix] Select the MXFP4 MoE runner for MiMo-V2 packed experts on SM100: #41668
- [MoE] Keep the moe_align_block_size pad fill inside sorted_token_ids: #41759
- [Fix] Shard MoE WNA16 and Quark INT4-FP8 weights by the MoE placement: #41807
- [EPLB] Warm default-group NCCL P2P transports before KV-cache sizing: #42026
- [Fix] Select the Marlin MoE runner for MiMo-V2 packed MXFP4 experts on SM90: #42172
- [MoE] Skip block-padding rows in the TMA down activation and quant: #42212
- [Fix] Offer the fused MoE finalize all-reduce only to a block that owes the sum: #42479
- [GLM-5.3-Flash] Autotune MoE kernels at prefill sizes: #42693
- [MoE] Fill num_token_non_padded for the customized A2A backend at EP 1: #43245
- support DeepEP v2 on GLM-5.3: #43432
- [FlashInfer] Add startup autotuning for MegaMoE: #43789
Quantization
- [Qwen3.8-Flash-Next] Optional fp8 (e4m3) storage for the compressed QSA indexer cache: #39614
- Fix/compressed tensors nvfp4 hopper: #40022
- [JIT] Fuse FP8 KV-cache quantization into the prefix-valid commit kernel (2.99x faster commit kernel on B200, kernel level): #40785
- fix: route Thor SM110 FP8 and ModelOpt NVFP4 auto backends: #41765
- [DSA] Q8KV8 FP8 Sparse Prefill on GLM-5.3-Flash with BF16 KV Cache (sparse prefill 1.53x to 1.72x faster per DSA layer than tilelang on H200): #41790
- [DeepSeek-V4.1] mxfp8 dispatch cache: #42156
- [Fix] Guard DeepSeek NVFP4 shared-expert fusion for LoRA and FP4 backends: #42203
- [Fix] Quantize reloaded FP8 weights after native sharding: #42585
- perf(mla): use native SATFINITE FP8 conversion in MLA KV/Q preparation: #42659
- [Perf] Cache compiled ModelOpt exclusion patterns: #42927
- [DSA] Support TileLang FP8 KV cache on CUDA: #42957
- [Fix] Account for captured inputs in DeepGEMM masked workspace budget: #43563
Parallelism & Disaggregation
- [Fix][DP] Prevent health checks from perturbing user request routing: #35721
- [PD] Centralize drain-aware abort acknowledgements: #37077
- NIXL: Use stride desc API: #37827
- [PP+PD] Fix #38206 Reduce bootstrap consensus latency: #38959
- [Metrics] Fix PD latency histogram accounting: #39706
- [PD] Preserve bootstrap metadata in native Messages requests: #39711
- [Qwen3.8 CP] Context parallelism for QSA attention and the sparse indexer: #39721
- [KV Shard] Enable Control Plane C: #40102
- [DeepSeek-V4.1][PD] Decode node support DP attention: #40177
- [DeepSeek-V4.1][PD] Support prefill node use cp parallel strategy: #40323
- [PD] Add opt-in prefill-complete decode KV allocation (26.8% higher throughput and 48.9% lower p95 TTFT on GLM-5.2-NVFP4 under decode KV pressure): #40703
- [PD] Support heterogeneous TP for hybrid SWA KV transfer: #40909
- [Fix] Stop PD-decode queue_time from counting decode time before a retraction: #41380
- [PD] Wait for prefill completion before Mooncake early KV transfer: #41395
- [PD][Mamba] fix: free the COW mamba slot of decode requests dropped before preallocation: #41451
- [PD] Validate state strides before Mooncake transfers: #41607
- [PD] Reload NIXL peer metadata after the peer is invalidated: #41691
- [PD] Publish prefill DP rank whenever forced lookup is enabled: #41786
- [Fix] State the configured MoE-DP width in the weight-cache fingerprint: #41806
- [Fix] Size the prefill delayer's gather buffer to its TP group: #41809
- Add --attn-dp-size and deprecate --enable-dp-attention: #41818
- [PD] Fix block scale registration for mixed full/SWA KV dtypes: #41924
- [PD] Fix crash for allocation when mamba + pd + decode radix cache: #41927
- [PD][NIXL] Fix prepared dlists for KV entries of different lengths (DeepSeek-V4): #42005
- [PD] Keep ingesting requests while a prefill forward result is pending: #42035
- [CP] Support DP x TP x CP for attention with interleave CP strategy: #42038
- [CP] Support scattered interleave CP inputs with expert parallelism: #42041
- [CP] GLM-5.3-Flash Prefill CP implemented on layer boundaries (P50 TTFT 2% to 9.5% lower than TP4 on 4x B300): #42044
- [PD] Validate decode state layout once at registration: #42051
- [PD] Make the prebuilt last-token H2D copy non-blocking under overlap: #42113
- [PD] Stop reserving unallocated mamba spec-verify scratch in prefill KV sizing: #42649
- [Metrics] Add per-rank DP attention imbalance metrics: #42755
- [Fix] Return an FFN's output to the attention rows at a pipeline handoff: #42892
- [Fix] Shard Kimi K3's shared experts over the TP group they sum over: #42895
- [PD] Allow state-only KV checksum inputs: #42904
- [GLM-V] Shard the vision tower over the attention-TP group under DP attention: #42931
- [PD] Charge decode admission the page-rounded allocation when the radix cache is disabled: #43173
- [Metrics] Count per-rank DP attention pairs and their imbalance: #43180
- [Metrics] Use widely supported bucket boundaries for the DP attention imbalance ratio: #43187
- [DeepSeek-V4.1] Enable DP attention with MegaMoE: #43228
- [Scheduler] Keep a DP-converted decode batch as the running batch: #43283
- [CUDA Graph] Size dp-local decode graphs by the runner's own gather requirement: #43303
- [PP+PD] Harden PPConsensusStore: msgspec wire format, drop malformed messages: #43329
- [PD] Use 1 to disable decode host receive and 0 for always-on: #43407
- [PD] Receive decode KV into separate paged SWA host pools; let host-capable receivers own abort drain tracking: #43410
- Skip the eager activation reserve in KV sizing on a capped Full prefill role: #43435
- [PD] Reclaim parked optimistic prefills when request slots are exhausted: #43442
- [Scheduler] Keep waiting-prefix LRU refresh consistent across TP ranks: #43479
- Preserve serialized page dtype in NIXL copy fallback: #43593
- Keep NIXL Mamba checkpoint keys separate by TP rank: #43594
- [PD] Poison the deferred abort ack when a Mooncake send fails: #43719
- [PD][NIXL] Fix staging buffer startup by accepting slot_layer_ids: #43723
- [DCP] Decode context parallelism for DSA (GLM-5.2 / GLM-5.3) with MTP (on GLM-5.3, 8x B300, adding MTP speeds DCP2 up 1.8x and DCP8 up 2.1x): #43735
- [PD] Retract transferred decode requests on a retract pause: #43757
- [DeepSeek-V4][CP] Write low-ratio page indices when CP prefill runs paged attention: #43935
- [PD] Recompute retracted DeepSeek-V4 requests on prefill instead of crashing decode: #43936
- [DeepSeek-V4.1] Skip Mega mHC fusion on 0-row (DP-idle) batches: #43993
- [PD] Return prompt routed experts from the PD prefill server: #44002
Scheduler & Runtime
- [Fix] overlap scheduler: record_stream mix_running_indices for the forward-stream relay gather: #34076
- [Platform] Thread vendor KV pool classes through the in-tree builders; add PlatformCapabilities: #35594
- [Metrics] Fix non-streaming TTFT by flushing the first output through detokenization: #38600
- [Weight Cache] Coordinate daemon readiness across ranks: #38828
- feat(grpc): expose follower metadata and node-local KV sources: #39659
- tokenizer perf optimization: #39771
- [Metrics] Add request-level TPOT histogram (sglang:request_time_per_output_token_seconds): #40275
- fix(nccl): disable graph buffer registration for pausable graph pools: #40648
- feat(metrics): expose deferred decode KV release metrics: #41128
- [Perf] Tune SM90 GDN recurrent verify launch for small batches: #41486
- [KDA] Fix ptx_kda prefill NaN without a gate lower bound and workspace growth: #41572
- [Fix] Release consumed residual contributions: #41749
- [Fix] scheduler: stop latching batch_is_full when nothing can run: #41872
- [Fix] Make tokenizer request-state initialization atomic: #41911
- [Metrics] Label scheduler stage wall time by sampled forward overlap: #42025
- [Perf] Reduce multi-detokenizer router IPC sends: #42238
- Pick the default nccl_port below the kernel ephemeral range: #42405
- [Fix] Make the fused KDA beta sigmoid bit-identical to torch.sigmoid: #42611
- [Scheduler] Gather prefill-delayer queue timeout across ranks: #42625
- Bind the MessageQueue remote socket atomically: #42695
- [Scheduler] Skip the decode for a request whose last token is already in flight: #42720
- [Metrics] Read scheduled prefill KV tokens from batch
prefix_lens: #42797 - [dLLM] Keep the request row across FDFO blocks: #42846
- [Perf] Fuse short-convolution checkpoint tracking metadata: #43013
- [Memory] Share one full prefill graph output buffer across shapes for tuple and list outputs: #43176
- Overlap scheduler startup with the data parallel controller: #43177
- [Perf] Avoid gate copies in gated RMSNorm: #43227
- [Perf] Fuse BF16 writes to native HND KV cache storage: #43237
- [Scheduler] Hold back decode rows on a mixed step when they do not fit the KV pool: #43290
- [Scheduler] Requeue preempted requests when admission admits nothing: #43298
- Fix torch.compile crash in fused gate-sigmoid-mul launcher: #43384
- [KV Events] fix missing topics in replayed kv events: #43386
- [Fix] Keep request-pool row generations monotonic across clear(): #43439
- [Scheduler] Fix the deterministic-inference hang when a prefill chunk is shorter than its alignment: #43444
- [Fix] Reserve remaining tokens for ignore_eos requests during prefill admission: #43564
- [Perf] Compact GDN causal-conv grids for skewed prefill batches: #43570
- [Fix] Adapt to transformers 5.18 / 5.19 changes: #43688
- [RL] Return a failed weight update when the cache flush is refused: #43758
- [Core] Add request-level customized_info keys and mark Full prefill capture batches: #43772
- [Observability] Report per-request storage prefetch and staging waits in meta_info: #43946
- [Fix] Work around a CPython fork deadlock in the CPU preprocess pool: #44143
Sampling
- Add Aaronson-Gumbel text watermarking: #37577
- [Kernel] Use Cake softmax for qualified SM103 sampling workloads: #42004
- [Debug] Add opt-in NaN detection before sampling: #42690
- Return output top logprobs as flat base64 arrays on request: #42890
- Inject "req" into custom_params only for custom logit processors: #43181
- [Sampler] Sort only the max(top_k) leading columns in the torch sampler: #43346
HiCache & Radix Cache
- [HiCache] Route --file-storage-path to the file storage backend: #33883
- [MemCache] Bound FlexKV checkpoint stores and restore DSPARK cache checks: #35694
- [Mamba] Warm cache COW kernel before serving (first-wave cold excess 924 to 28 ms): #36266
- [HiCache] Support Mamba in buffer-only host memory mode: #36345
- Allow platforms to declare Mamba extra-buffer support: #36410
- [Unified Memory] Speculative decoding on the unified memory pool: translate every read and write path: #37627
- [Unified Memory] Fuse the draft model's KV into the target's page envelope (mechanism): #37628
- fix: wait for HiCache load-back before hybrid MLA KV reads: #38217
- fix(inkling): translate unified-memory checkpoint destinations to physical slots: #38229
- [HiCache]: Fix host lock ownership across radix splits: #38480
- [Unified Memory] Token-major dense views for the unified memory pool (3/7): #38592
- Fix unified HiCache physical transfers: #39479
- [HiCache] Add staged write-back for the page-unified KV cache layout: #39606
- [Perf] Batch Mooncake hybrid-pool metadata and puts: #39696
- [Fix] HiCache: carry registered Mamba slot side states (Qwen4-Exp PLE) through the host tier: #39862
- fix: preserve QSA indexer state through HiCache: #39893
- [Fix] Fix unified-memory compaction gates and pending page reuse: #39982
- [HiCache] Preserve all MTP KV and recurrent state: #40223
- [Unified Memory] Derive KV row addresses from strides, not shapes (1/7): #40326
- [Unified Memory] Build paged KV views through one helper (2/7): #40327
- [Unified Memory] Mark the write loc physical and check it at every write door (4/7): #40328
- [Unified Memory] Remove the kernel-page multiplier plumbing (5/7): #40329
- [Unified Memory] Stride the KV translate kernel and route every translate through it (6/7): #40330
- [Unified Memory] Name the fused KV translate for what it computes (7/7): #40331
- fix(dsv4): ship the fp8 unified_kv rope pool through the direct external linker: #40471
- [Unified Memory] Admit EAGLE/EAGLE3 onto fused draft KV, on mamba-MHA and MLA hosts: #40491
- [Unified Memory] Admit DFLASH and DSPARK with fused draft KV: #40492
- [Unified Cache] Abort requests whose external linker load fails instead of crashing the scheduler: #40896
- Declare HiCache host pools for DSA indexer: #40913
- Declare hybrid Mamba indexer host pool: #40915
- [Fix] Allocate SWA KV pools inside the memory-saver region: #40979
- [DeepSeek-V4] Reserve the FULL logical page in the c4 indexer pool like its KV pool: #41163
- [HiSparse] fix: return HiSparse slots on PD decode waiting abort and DeepSeek V4 free: #41346
- [HiCache][PD] fix: keep decode load-back polling out of rank-local collectives: #41393
- [HiSparse] Avoid host synchronization in eager backup: #41446
- [HiCache][PD] fix: clamp the decode restore to the prefix promised to prefill: #41450
- [MemCache] Rename
cache_unfinished_reqtocheckpoint_reqand count cache hits only when a request finishes: #41520 - [Scheduler] Refresh waiting requests' cached prefixes under FCFS-like policies with LRU eviction: #41593
- [HiCache] Attribute buffer-mode storage hits against the joint device match: #41758
- [HiCache][PD] Allow DSA index-K elision under HiCache and disaggregation (KV capacity +11.6% on prefill and +13.7% on decode for GLM-5.2-NVFP4 PD): #41769
- [Fix] Read TensorCast's WORLD placement under its current names: #41805
- Support post-capture KV sizing for the unified hybrid-SWA pool: #41961
- fix(unified-memory): propagate unified-memory lazy checkpoint policy and handle unallocated session slots: #42072
- [HiCache] Avoid full-store scans for uncached file prefix lookups: #42077
- [HiCache] Fall back to host memory when no cgroup fs is mounted: #42120
- [Fix][DeepSeek-V4.1] SWA page size with bounded replay: #42128
- [HiCache] Let the MiniMax-M3 K-only host pool join a host pool group: #42166
- [DSA] k-pool: 256-token logical page so index page id = logical page id: #42178
- [MemCache] Skip the release-time insert on optimistic prefill requeue; make
refresh_fill_idspublic: #42194 - [HiCache] Fix DeepSeek-V4 storage backend crash from missing storage_format_tag: #42215
- [Unified Memory] Copy page envelopes in place during compaction: #42252
- [HiCache] Fix write-back SWA insert backups tripping the write-through pending-ack assert: #42264
- [Session] Run streaming sessions only on
UnifiedRadixCache; reject unverified tree caches: #42295 - [Session] Release aborted streaming turns through the normal release path: #42330
- [MemCache] Run mamba models on
UnifiedRadixCachewhen the radix cache is disabled: #42354 - [HiCache] Add SeaweedFS L3 storage backend (zero-copy path cuts L3 TTFT 3.5x to 4.7x for Qwen3-4B on one A10): #42399
- [HiCache] Fix cgroup page-cache accounting and the sizing fallback when cgroup discovery fails: #42420
- [Fix] Make Hf3fsMockClient reads and writes thread-safe with pread/pwrite: #42425
- [MemCache] Keep prefix_indices current for chunked requests kept out of the tree: #42467
- [Session] Let a streaming session own its KV record and tree lock from the first turn: #42537
- [Session] Count streaming session KV by owner in pool accounting: #42539
- [HiCache] Manage buffer-mode backups per storage pool: #42633
- Rename
SGLANG_OPT_SWA_RELEASE_LEAF_LOCK_AFTER_WINDOWtoSGLANG_OPT_RELEASE_PREFILL_SWA: #42657 - [HiCache] Resolve write_back to write_through under buffer_only host memory: #42754
- [HiCache] Keep protected cgroup page cache charged in host budgets: #42788
- [MemCache] Return only the matched length from
match_prefix; read KV indices off the node path: #42923 - [MemCache] Return only the loaded length from
init_load_back; dropempty_device_indices: #43023 - Fix DSA index host buffer cleanup: #43100
- Fix K-only host buffer cleanup: #43101
- [DeepSeek-V4.1] Initialize the full-to-SWA mapping for request-window pools: #43165
- [HiCache] Re-admit a split node's queued buffer-mode write instead of dropping it: #43174
- [Session] Skip the KV free when page alignment empties a streaming-session rewind: #43183
- [MemCache] Count reclaimable tokens in the unified tree ledgers: #43185
- [MemCache] Serve disabled pure-SWA models with
PureSWARadixCache: #43198 - [MemCache] Serve disabled full-attention models with
UnifiedRadixCache: #43199 - [MemCache] Serve disabled hybrid-SWA models with
UnifiedRadixCache: #43200 - [MemCache] Delete
ChunkCache: #43201 - [MemCache] Skip SWA request-cap pool sizing when chunked prefill is off: #43203
- [Unified Memory] Allocate without transfer waits, pin floating SWA reservations, drop the HiCache startup asserts: #43206
- [MemCache] Skip KV events and eviction config on a disabled radix cache: #43253
- [HiCache] Keep storage workers alive on backend exceptions: #43550
- [PD][HiCache] Fold the decode restore outcome into the MIN-reduced KV poll: #43562
- [Unified Radix] Invalidate lazy Mamba checkpoints before in-place tracking: #43567
- Share MLA host capacity initialization: #43595
- Build Mamba host pools from device buffer infos: #43596
- Capture KV host transfer buffers: #43711
- [HiCache] Block-group pool hits, caller-owned load destinations, allocation-target FULL eviction: #43868
- [Unified Memory] Prefer SWA reclaim quotas with a HiCache storage backend or on PD decode: #43919
- [Core] Keep request priority within the signed 64-bit range in radix caches: #43920
LoRA
- Fix LoRA usage-counter accounting across request lifecycle paths: #31808
- [LoRA] Support LoRA for the NemotronH_Nano_VL_V2 model family: #39798
- [Perf][LoRA] remove empty CTAs from fused LoRA expand and add split-K to shrink: #41951
- [KV Events] Publish LoRA adapter names and namespaced block hashes: #42455
Multimodal
- fix(mimo-vl): keep multimodal features when capturing aux hidden states, and correct vision preprocessing: #40888
- fix(multimodal): convert grayscale tensors to RGB in PIL backend: #42127
- Fix EVS for NemotronH_Omni_Reasoning_V3: #42933
- [Multimodal] Chunk and cache audio items whose offsets span several placeholder runs: #43301
- [Fix] Repair multimodal VMM lease cleanup and source-stream ordering: #43475
Model Support & Optimizations
- [dLLM] Add JointThresholdInDel algorithm for insertion & deletion decoding: #31773
- [Fix] Fix DeepSeek V4 Pro TP24 vocabulary-padding failure on Hopper: #31801
- [Kimi] Enable GB300 TP4 and GB200/GB300 TP16 SP collectives: #35330
- [Fix] Pass token-major Q/K tensors from Gemma-3 to RadixAttention: #37984
- [MiniMax-M3] Instantiate the decode block top-k in register buckets: #38615
- [MiniMax-M3] Decode: paged K/V tile loads and a tiny GEMM for the router projection: #38841
- [Qwen3.8 CP] Collocated prefill CP integration: #39723
- [Kimi-K3] Guard optimized paths by platform: #40269
- [Model Loader] Add InstantTensor loader support (DeepSeek-R1 on 8x H200 loads in 15.3 s instead of 160 s from 50 GB/s storage): #40453
- [Fix] MiMo-V2: accept checkpoints with the split attention layout: #40978
- [Perf] Optimize DeepSeek V4.1 Flash Hopper paths and Blackwell prefill selection (2.1x to 2.3x output throughput per GPU on 8x H200): #41251
- [KDA][Kimi-K3] Speed up LTX-2.3 QK norm and split RoPE on H200: #41459
- [DeepSeek-V4.1] Fold q_rope_store into fused_q_norm_rope: #41657
- [DeepSeek-V4.1] Faster fp4 index-K gather and combine_topk_swa_indices: #41658
- [DeepSeek-V4.1] Fused c1/c2 compress for eager extend, faster c2 decode: #41660
- [Fix] Keep MiMo-V2 processor available without TorchCodec: #41667
- [DeepSeek-V4.1] Trim prefill model-entry overhead: #41704
- [Fix] Reduce the Sarvam dense MLP through its row-parallel projection: #41752
- [Fix] Step-3.5: build the shared expert without TP under all-to-all MoE backends: #41754
- [Refactor] Build seven more decoders from stage boundaries: #41779
- [Fix] Make the Solar model constructible and runnable: #41810
- Add blackwell dual gemm fusion support for MLP: #41887
- [DeepSeek-V4.1] Reduce eager Engram extend preparation: #42159
- [Perf] Optimize Qwen3.8-Flash-Next BF16 decode on Blackwell (242 to 271 tok/s on 4x B200 at concurrency 1): #42175
- Serve pplx-decider decision checkpoints on /v1/systemone: #42183
- [Fix] Validate per-layer rope_parameters on transformers 5.17 (Laguna-S/XS-2.1 nightly): #42214
- [Perf] Accelerate DiffusionGemma with FA4 and CUDA graphs on Blackwell (7.6% higher generation throughput): #42239
- [DeepSeek-V4.1] Unify mHC into one state machine and drop medium-batch fusion variants: #42245
- [Fix] Step-3.5 DeepEP routed scaling and Sarvam shared expert under dense TP1: #42302
- [Fix] EXAONE and ERNIE 4.5 VL MoE architecture, backend and PP issues: #42303
- [Fix] EXAONE MoE under DP attention and DeepEP: #42305
- [Fix] Jet-Nemotron build and Granite MoE hybrid final norm: #42306
- [Refactor] Build the Qwen2 decoders from stage boundaries: #42307
- [Refactor] Build the Llama and Nemotron-NAS decoders from stage boundaries: #42308
- [Fix] GigaChat 3.5: apply the sandwich norms to the complete sums: #42310
- [Refactor] Build the ZAYA1, IQuest-Q1 and Gemma 4 decoders from stage boundaries: #42311
- [Refactor] Build the Hunyuan V4 decoder from stage boundaries: #42477
- [Refactor] Build the Qwen4 experimental decoders from stage boundaries: #42478
- [Fix] GigaChat 3.5: build the stage boundaries once: #42480
- Serve pplx-decider-v1.1 and update the pplx-decider cookbooks: #42645
- [Model] Support Clef and Clef-Flash decision models on /v1/systemone: #42721
- [Model] Add support for EmbeddingGemma 2: #42782
- [DeepSeek-V4.1] Mega-mhc integration: #42808
- [Fix] Kimi K3 KDA value head size and SiTU input alignment: #42893
- [Refactor] Run Kimi K3 through stage boundaries: #42894
- Fix Mistral-native format detection when the Hub is unreachable: #43446
- [Fix] Keep residual rows and sums consistent across appends and pipeline handoffs: #43454
- [Model] Keep gpt-oss router logits batch-invariant under deterministic inference: #43624
- [DeepSeek-V4.1] Support decoder SWA bounded replay with CUDA DP attention: #43812
- [DeepSeek-V4.1] mHC: drop seam_open, narrow precomputed stats in take_rows: #43833
- [DeepSeek-V4.1] Reduce sparse-prefill workspace padding: #43921
- Reuse Hugging Face LFM2-MoE configuration: #43939
Kernel Library
- Add triton autotune on the Mamba2 SSD kernels: #39130
- [KDA] Add an opt-in decode-parity mode to the Triton multi-token recurrence: #42251
- Update AOT kernels for Torch 2.14: #42366
- [Kernel] Use explicit reciprocal scaling for recurrent Q/K normalization: #42486
- [SM90] Use native block32 DeepGEMM for DeepSeek-V4.1: #42604
- [Fix] Guard merge_state sgl_kernel import on non-CUDA platforms: #42682
- [Kernel] SwiGLU-OAI MXFP8 producer; move swizzled MXFP8 producers to ops/quantization: #42810
- [DeepSeek-V4.1] Use num_stages=1 for the hc_mix stats partial kernel: #43070
- [Mamba] Describe the track snapshot dtype by its kernel instead of fp32: #43192
- [Perf] Enable fast-math for JIT activations on Blackwell: #43236
- [GDN] Fix CuTe DSL decode to read the [N, HV, V, K] state layout: #43332
- [Fix] Link cuda_ipc JIT against CUDA driver stubs dir: #43764
OpenAI-Compatible API
- [Responses] Fix max_output_tokens overestimate on multimodal requests: #35524
- [Score] Send stacked GPU embedding overrides through a CUDA IPC pool: #39730
- [Fix] Preserve inline instructions for Responses and Kimi K3 Messages: #40077
- [Frontend] Parallelize long chat prompt encoding: #41259
- [Fix] Emit Responses reasoning content-part events: #41789
- Advertise the KV-event replay endpoint in /server_info: #42274
- [Fix] Preserve empty K2 Horizon reasoning in chat responses: #42623
- Stream a chunk once stream_interval tokens are unsent: #42643
- [gRPC] Move runtime.v1 bindings to tonic 0.14, report OpenAI status codes, return text on request: #42673
- [Perf] Serialize OpenAI chat/completion responses in one pass: #42889
- fix(openai): preserve zero cached tokens when cache reporting is enabled: #43005
- fix(openai): honor tool_choice=none in DeepSeek encoders: #43006
- fix(openai): keep chat stream created timestamp stable: #43026
- fix(openai): omit absent tool_calls from chat responses: #43027
- [HTTP Server] Let custom /generate routes skip the second contract check: #43114
- [Fix] Preserve HTTP errors for non-streaming /v1/messages: #43796
Rust Server
The experimental Rust router and frontend (experimental/sgl-router, sglang-processor) keep moving toward parity with the Python server.
- [SGL Router] Watch EndpointSlices for sibling router replicas (4/13): #40690
- [SGL Router] Track a booting rank's bootstrap and fetch a peer's snapshot (5/13): #40691
- [SGL Router] Vet a peer's snapshot before it may touch the tree (6/13): #40692
- [SGL Router] Hold a booting rank's batches and graft a snapshot on the pump (7/13): #40693
- [SGL Router] Sweep the fleet for a peer snapshot and hand it to the pump (8/13): #40694
- [SGL Router] Share one fleet-wide fetch across a discovery burst (9/13): #40695
- [SGL Router] Ask the fleet when a graft's splice goes unwitnessed (10/13): #40696
- [SGL Router] Gate readiness on peer bootstrap, and make it observable (13/13): #40699
- [SGL Router] Add --tokenizer-backend fast and --tokenizer-l1-cache-mb: #41268
- [SGL Router] Exclude portless prefills and retry bootstrap discovery: #41610
- [SGL Router] Allow load-only routing without a tokenizer: #41611
- [SGL Router] Fail fast and cancel decode when prefill fails: #41612
- [SGL Router] Enforce compatible PD version groups: #41613
- [Rust] Define semantic frontend contract: #41709
- [Rust] Extract shared runtime.v1 protobuf bindings: #41726
- [Rust] Implement the Rust gRPC adapter for native generation: #41766
- [Rust Renderer]
sglang-processorlib: #41896 - [Rust] Wire the gRPC server into the frontend lifecycle: #41964
- [SGL Router] Route to a specific DP rank inside multi-rank workers (--dp-aware): #41974
- [SGL Router] Unify reorg session and cache affinity modes: #41990
- [SGL Router] Support engines started with --api-key (--worker-api-key): #41996
- [SGL Router] Harden peer-bootstrap edges: refuse redirects and metric docs: #41999
- [SGL Router] Drop the snapshot client's redirect-following fallback: #42010
- [SGL Router] Add native /generate endpoint: #42147
- [SGL Router] Add OpenAI /v1/embeddings: #42191
- [SGL Router] Add SGLang's /v1/rerank: #42205
- [SGL Router] Add SGLang's /v1/classify: #42258
- [SGL Router] Repair KV-event sequence gaps from the engine's replay socket: #42275
- [SGL Router] Credit routed prompts before their KV events arrive: #42276
- [SGL Router] Serve the Anthropic Messages API on /v1/messages: #42316
- [SGL Router] Move pressure and prefix signals out of legacy policies: #42428
- [SGL Router] Add reorg bucket config with per-group admission: #42429
- [SGL Router] Improved policy balanced mode: #42545
- [SGL Router] Let selection skip workers a request already failed on (retry 1/5): #42628
- [SGL Router] Make one request dispatchable more than once (retry 2/5): #42629
- [SGL Router] Retry failed dispatches on another worker (retry 3/5): #42630
- [SGL Router] Back off between retry attempts (retry 4/5): #42631
- [SGL Router] Bound streaming time-to-headers by --request-timeout-secs (retry 5/5): #42632
- [Rust Processor] Split sglang-processor into tokenizer, render and parser: #42662
- [Rust Processor] DeepSeek-V4 parity with SGLang, plus a reusable parity harness and skill: #42664
- [SGL Router] Render DeepSeek-V4 through sglang-processor: #42665
- [Rust] Update Dynamo frontend crates to latest compatible releases: #42666
- [SGL Router] Document and reject a zero request timeout: #42751
- [Rust] Upgrade sglang-radix-tree to PyO3 0.29: #42807
- [Rust Server] preserve PD routing in chat and completions: #42989
- [Rust Processor] Share the reasoning splitter and tool constraint across hosts: #42996
- [SGL Router] Serve /v1/completions through /generate, with the shared OpenAI layer: #42997
- [SGL Router] Serve /v1/chat/completions through /generate, with reasoning parsing: #42998
- [SGL Router] Parse tool calls when serving chat through /generate: #42999
- [SGL Router] Serve DeepSeek-V4.1 Flash through sglang-processor: #43000
- [Rust Server] warm up each rust listener in-process: #43004
- [Rust] Pass the media item fps hint to multimodal processors: #43143
- [SGL Router] Keep charset=utf-8 on streamed responses: #43232
- [SGL Router] Read finished /generate streams to the end; expose reorg admission caps as flags: #43233
- [Rust] Serve any
modelname on the OpenAI chat and completions routes: #43923 - [Rust] Let an embedding package wrap the frontend HTTP app: #43924
- [Rust] Refuse an unsupported multimodal model before the Rust listener binds: #43925
Simulator
- [Bench][MoE] Deal the simulated round-robin experts evenly across EP ranks: #43188
- [DSpark] Apply
SGLANG_SIMULATE_ACC_LENinside the accept instead of switching paths: #43623 - [DFlash] Decide the Domino candidate pool in one place and apply simulated acceptance inside the accept: #43791
SGLang-Diffusion
- [Diffusion] fix: honor the lazy transformer loader contract: #15215
- fix(diffusion): validate output_quality and fix return type annotation: #19530
- [Diffusion] Cache LTX-2 RoPE coords to avoid per-step recompute: #22441
- Fix scheduler host not working when ipv6 in multi modal gen: #22813
- fix(zimage): align SGLD CFG combine formula with diffusers Z-Image: #23772
- [Diffusion] Enable channels_last_3d for Cosmos3 VAE: #28021
- [Diffusion] Support Ideogram TurboTime LoRA inference: #30487
- [Diffusion] attention: Allow disabling sequence masking in SP: #33855
- [Diffusion] support MiniMax H3 RL: #34365
- [Diffusion] Forward multimodal token type IDs in image encoding: #34494
- [Diffusion] cli: restore gated Diffusers model detection: #34903
- [Diffusion] Fail MiniMax-H3 quality="high" closed on substituted DiT weights: #34978
- [Diffusion] Fix width/height silently dropped on multipart /v1/videos requests: #35108
- [Diffusion] Allow per-role attention backend override: #35310
- [Diffusion] Z-Image: autotune the shared transformer block at both token scales (H100 1024^2 denoise 13.58 -> 9.58 s, 1.42x): #35755
- [Diffusion] Add configurable reference image resizing for MiniMax H3 Ref2VA: #35839
- [Diffusion][MiniMax-H3] Add community LoRA recipes and Kohya mapping: #35857
- [Diffusion] Add support for HunyuanImage-3.0-Instruct: #36311
- [Diffusion] feat: enable breakable CUDA graphs for FLUX.2 Klein: #36470
- [Diffusion] feat: add --async-output-save to overlap output saving with the next request: #37549
- [Diffusion] perf: FastH3 8-Step V2 native support, native Blackwell VSA, batched H3 VAE decode: #37662
- [Diffusion] Support LLaDA-Image BF16 generation and editing: #37907
- [Diffusion] fix: do not interleave comfy-kitchen MXFP8 scales twice: #38028
- [Diffusion] quantization: ConvRot INT8 online W8A8 for DiTs (sgl-kernel +
convrot_int8_customkernel): #38040 - [Fix] MiniMax H3: honor requested inference step count: #38671
- [Diffusion] model: restore folded H3 GGUF patch embedding: #39523
- [Diffusion][MiniMax-H3] Avoid full-video clone in VAE postprocessing (drops a 4.2 GiB full-video copy, postprocessing about 21% faster): #40095
- fix: resolve QKV layout inference for MiniMax-H3 checkpoints: #40158
- [Diffusion] USP: accept strided row views in the packed QKV all-to-all: #40588
- [Diffusion] Skip warmup preferred preload when it would OOM: #40761
- fix: wait for multimodal process teardown: #40765
- [Diffusion][Model] Add native SANA-Video 2.0 support: #41492
- [Diffusion] fix: validate SageAttention 2 head sizes without overriding component policy: #41529
- [Diffusion] Deduplicate SM120 FP8 fallback warnings across layers: #41542
- [Diffusion] Respect explicit residency over pipeline preload hints: #41622
- [Diffusion] Keep generic warmup valid for step-floored models and stop reporting a failed warmup as warm: #41641
- [Diffusion] Fuse Flux3 rowwise FP8 quantization with Triton: #41671
- Fix CFG normalization to use per-sample norm instead of batch-flattened norm: #41706
- [Diffusion] kernels: stop specializing on per-request sequence lengths: #41710
- [Diffusion] perf dump: one writer per replica, no first-request NCCL setup: #41711
- [Diffusion] release the allocator cache when the server idles, not per request: #41718
- [Diffusion] Stream an oversized DiT in auto mode instead of OOMing: #41721
- [Diffusion] Keep the DiT off component offload under explicit multi-GPU FSDP: #41722
- [Diffusion] Fix the OOM hint's placement advice and add a 24 GB Wan2.1 recipe: #41723
- [Diffusion] video: let requests choose the libx264 preset: #41797
- [Diffusion] Encode the MP4 while the VAE decodes (MiniMax-H3, Wan) (Wan2.2 TI2V 5B request 6.27 to 5.83 s on one B200): #41819
- [Diffusion] H3: check the final MP4 in-process instead of spawning ffprobe: #41825
- [Diffusion] kernels: take the Wan VAE decoder's remaining first-decode work off the request path: #41830
- [Diffusion] conditioning cache: recycle warmup-only entries at the first served store: #41832
- [Diffusion] warm the image writer and the HTTP route models at startup: #41833
- [Diffusion] Action API: decode JSON pixel lists to numpy at the entrypoint: #41895
- [Diffusion] MiniMax-H3 fuse the video VAE decoder's RMSNorm and QK RoPE (10 s clip VAE decode 9.19 to 8.25 s on one H200): #41906
- [Fix][Diffusion] Fix hybrid SP+TP for I2V models: #41917
- [Diffusion][MiniMax-H3] Fix int32 offset overflow in SubBlock router kernels: #41965
- [Diffusion][MiniMax-H3] Route SubBlock sparse attention in head chunks: #41985
- fix(diffusion): load serialized H3 INT8 in ComfyUI integrated mode: #42121
- [Diffusion] FLUX 3 Action: CUDA graphs for observation encoding and denoising steps: #42171
- [Diffusion] Qwen-Image 2.1: load ComfyUI/ai-toolkit fused
gate_upLoRAs and diffusers metadata alpha: #42174 - [Diffusion] fix: keep residual_gate_add on the JIT CUDA path for cont…: #42240
- [Diffusion] Fix native FP8 format handling for FLUX 3 rowwise linears: #42255
- [Diffusion] Name the quality tiers after what they guarantee: #42370
- [Diffusion] Fix TeaCache CFG state lifecycle and skip boundaries (skips 30% of DiT forwards, 1.35x on one GPU): #42422
- [Diffusion] Reset DiT cache state at the start of each request: #42437
- [Diffusion] fix: pin the JoyAI-Echo overlay's source revision: #42471
- [Diffusion] fix: load community ComfyUI-GGUF Qwen-Image-2.1 transformers: #42556
- [Diffusion] Fuse Anima AdaLN modulation and cache the FFI callable: #42608
- [Diffusion] Fuse Anima FP32 split-half Q/K RoPE (Anima denoise 2.84 to 2.25 s on one H200): #42609
- [Diffusion] model: support Kandinsky 6: #42743
- [Diffusion] Skip zero blocks when hashing large conditioning inputs: #42748
- [Diffusion] Hunyuan3D: implement paint_use_remesh (decimate to 40k faces before UV unwrap): #42770
- [Diffusion] model: support Wan-Animate-2: #42941
- [Diffusion] avoid redundant LoRA scaling while preserving exact numerics: #43028
- [Diffusion] Add ModelOpt FP8 checkpoint quantization for Pi0.5: #43062
- [Diffusion] Don't report video jobs completed after their outputs are deleted: #43084
- [Diffusion] perf: decode plain base64 MiniMax-H3 materials without a per-character loop: #43117
- [Diffusion] Skip LTX-2 SP masks only for globally unpadded sequences: #43134
- [Diffusion] Fix int32 offset overflow in the Ulysses packed-QKV pack kernel: #43140
- [Diffusion] perf: pipeline MiniMax-H3's Ulysses exchange against attention over the copy engine: #43164
- [Diffusion] comfyui: report the real runtime import error: #43178
- [Diffusion] perf: tile MiniMax-H3's indexed modulation kernels by 2048 columns: #43267
- [Diffusion] fix: share the IPC all-to-all transport between AllToAll4D and USP: #43269
- [Diffusion] perf: fuse MiniMax-H3's RMSNorm and indexed AdaLN under quality lossless: #43327
- [Diffusion] perf: run MiniMax-H3's QK-norm + RoPE two rows per warp, bit-identical: #43337
- [Diffusion] perf: move MiniMax-H3's pipelined Ulysses copies off the SMs: #43361
- [Diffusion] perf: pipeline MiniMax-H3's Ulysses exchange against cuDNN SDPA on Blackwell: #43373
- [Diffusion] fix: keep a LoRA unmerged under merge_mode auto when merging would round its update away: #43385
- [Diffusion] Support Qwen-Image 2.1 Turbo and checkpoint sampling grids: #43391
- [Diffusion] comfyui: consolidate FluxAdapter correctness fixes: #43490
- [Diffusion] comfyui: consolidate executor conditioning-cache fixes: #43492
- [Diffusion] comfyui: fail early instead of silently dropping features in SGLD integrated mode: #43495
- [Diffusion] Add the dynamic LoRA delta inside the second GEMM: #43501
- [Diffusion] comfyui: run MiniMax H3, FastH3 and VDN in SGLD integrated mode: #43504
- [Diffusion] Run head_dim-128 QK-norm + RoPE two rows per warp for every RoPE layout: #43508
- [Diffusion] Pipeline every USPAttention Ulysses path's exchange over the copy engine: #43521
- [Diffusion] Bind each GPU worker to its GPU's NUMA node: #43565
- [Diffusion] BCG: warn instead of disabling on unvalidated models: #43568
- [Diffusion] Accept request-level profiling on the image API: #43573
- [Diffusion] FLUX: declare regional compile regions: #43575
- [Diffusion] Qwen-Image-2.1: skip QKV packing while layers are offloaded: #43576
- [Diffusion] torch.compile: keep Cosmos3's fused kernels in the graph: #43577
- [Diffusion] comfyui: run Qwen-Image 2.1 in SGLD integrated mode: #43581
- [Diffusion] IPC all-to-all: keep exported buffers out of expandable segments: #43584
- [Diffusion] torch.compile: keep Qwen-Image-2.1 and Ulysses graphs whole: #43586
- [Diffusion] Keep the overlay probe from stalling cached startup: #43590
- [Diffusion] torch.compile: skip inductor's saved multi-kernel pick (Z-Image warm starts crash or hang): #43601
- [Diffusion] torch.compile: keep the TP all-reduce inside the graph (Qwen-Image-2.1 TP2 compile 2.2 to 1.5 s on GB300 with
--offload-during-compile false): #43602 - [Diffusion] torch.compile: serve whole graphs after offload-during-compile: #43631
- [Diffusion] fix: Qwen-Image BCG prompt padding misses every graph and attends the pad: #43632
- [Diffusion] fix: replay Qwen-Image-2.1 BCG graphs across prompt lengths: #43633
- [Diffusion] MiniMax-H3: run the attention section eager under torch.compile: #43634
- [Diffusion] Encode LTX video/audio through the shared bounded MP4 path: #43656
- [Diffusion] Speed up LTX-2 guidance and video output: #43667
- [Diffusion] perf: layerwise offload for the LTX-2.5 Gemma-4 text encoder: #43683
- [Diffusion] split and batch LTX-2.3 one-stage guidance passes under CFG parallel: #43684
- [Diffusion] fix: shard Qwen-Image adaLN modulation and text FFN under TP: #43691
- [Diffusion] bind worker CPUs to the GPU's NUMA node when memory policy is not permitted: #43756
Local & Desktop AI
- [Apple Silicon] Add an Apple Silicon (MPS) platform to the standard Torch model runner: #36780
- [MLX] Fix startup after logical token capacity change: #41314
- [MLX] Drain in-flight jobs before a retract-mode
pause_generation: #43282 - [MLX] Drop a retracted request's runner state at its re-prefill: #43284
- [MLX] Sync decode KV to the pool once at release, clamped to the owned prefix: #43292
AMD / ROCm
- dsv4.1-amd: serve DeepSeek-V4.1 on gfx950 (DSpark up to 1.9x the throughput of plain decode on 4x MI350X; 128K-input TTFT 860 to 576 ms): #41308
- [AMD][DeepSeek-V4.1] Route low-ratio indexer and candidate-block top-k through top-k v2 (top-k 1.4x to 2.6x faster at decode shapes): #41947
- [AMD][DeepSeek-V4.1] Switch the fp8 dense GEMMs on gfx950 to aiter's MXFP8 GEMM: #41970
- [AMD][DeepSeek-V4.1] Pick the gfx950 wo_a batched gemm tile by row count: #41971
- [AMD][DeepSeek-V4.1] Fix shared-expert fusion accuracy and speed up MoE routing on ROCm (GSM8K 0.891 to 0.906 with fusion): #42011
- [AMD][DeepSeek-V4.1] Build DSpark draft metadata inside the CUDA graph on ROCm: #42014
- [AMD][DeepSeek-V4.1] OPUS sparse prefill on gfx950 through layout conversion: #42017
- [AMD][DeepSeek-V4.1] Fuse MXFP8 activation quant into producer kernels on gfx950 (per-step GPU time 9.46 to 8.88 ms on MI355X TP2 with DSpark): #42055
- [AMD] Fuse prefill index-Q RoPE fake-quant and fp4 quantize kernels in DSV4.1: #42911
- [AMD] Add flydsl mxfp8 dense gemm kernel for dpsk v4.1 prefill: #43803
- [AMD][DeepSeek-V4.1] Fix DP attention + TP MoE decode NaN and support bounded replay (102.6K tok/s/GPU on AgentX at concurrency 32, 2x MI355X): #43077
- [AMD] Set maxseq length limit for DSV4.1 BCG metadata buffers: #42570
- [AMD][DeepSeek-V4.1] Fix wo_b input flatten for packed fp8 activations on gfx950: #42796
- [AMD] Add engram host/device address mapping: #42552
- [AMD] Update aiter version for v41: #42023
- [AMD] Support Quark DeepSeek-V4-MXFP4 checkpoints on the DSV4 stack, enabling shared-expert fusion and MTP (also fixes a loading bug that dropped about 25% of expert FFN channels): #31054
- [AMD] Port CP V2 to the DeepSeek-V4 HIP backend (mean TTFT 16% to 31% lower at C16 to C48, TPOT 2% to 12% higher): #34200
- [AMD][DeepSeek-V4] feat: enable PD-disagg with fp8 unified_kv on gfx950: #39166
- [AMD][DeepSeek-V4] feat: enable PCP with fp8 unified_kv on gfx950 (1.55x KV tokens, up to 15.8% higher throughput): #39923
- [AMD][MORI] Add MORI EP V2 support: #40666
- [AMD] Keep aiter's tuned GEMM for the DeepSeek-V4 compressors: #41931
- [AMD] Fix AITER weight slicing for CP decode TP: #39950
- [AMD][GLM-5.3-Flash] Enable FP8 and MXFP4 serving on gfx950 (+14.3% geomean output throughput on AgentX, 4x MI355X): #39273
- [AMD] GLM-5.3-Flash: fuse shared expert and KDA projections on Quark MXFP4 (median ITL up to 13.6% lower): #41870
- [AMD] Let the GLM DSA NextN draft declare its own shared-expert fusion architecture: #41258
- [AMD] Eliminate remaining gfx95 FP8 scale relayout copies: #41030
- [ROCm] topk v2: split one long row across blocks, the CDNA cluster-path equivalent (median ITL 18.7% lower at 600K context): #39931
- [ROCm] GLM-5.2: gfx950 four-kernel fused DSA indexer decode path (ITL 5.3% lower): #38583
- [ROCm] GLM-5.2 decode path: decode-shaped MoE/MLA tiles, split speculative softmax, and bf16 GEMM routing (+5.7% end to end): #41725
- [ROCm] Use the fused MLA absorb + RoPE + KV-write kernel for decode-sized forward modes only: #41533
- [ROCm] Eliminate HiCache migration stalls and use exact-size host allocations (P90 interactivity +3.7% for GLM-5.2 MXFP4 on MI355X): #42553
- [AMD] Fix Load and Inference of MLA models with Quark PTPC FP8 attention on ROCm (+7.8% output throughput on GLM-5.2-Quark-MXFP4-AttnFP8): #28734
- [AMD] Resolving bug in per batch size graph capture due to multiple capture runs per batch size: #39840
- [AMD] MiniMax-M3: fuse sparse QK norm, RoPE and cache writes with AITER (up to 6.8% higher throughput): #35357
- [AMD] Use AITER ASM prefill for MiniMax-M3 HD128 attention (+16.6% P90 interactivity, +8.4% throughput): #41707
- [AMD] Add opt-in MiniMax-M3 TP4 indexer context partitioning (2.64x faster indexer chain at 128K context): #41488
- [AMD] MiniMax-M3 indexer CP: packed scoring for EAGLE verify rows: #42614
- [AMD] Fix MiniMax-M3 EAGLE3 verification on ROCm: #41497
- [AMD] Fix Quark W4A4 MXFP4 MoE clamped-SwiGLU on the AITER path: #34540
- [AMD] gfx950 small-batch MoE: expert-count gate for the small sort and FP8 block-scale small-M kernel (about 20% better interactivity at concurrency 1 and 4): #41982
- [AMD] One-launch small-M MoE router for Qwen3.5 on gfx950: #41133
- [AMD] Small-M W8A8 FP8 projection GEMM for Qwen3.5 AttnFP8 on gfx950: #41134
- [AMD] Qwen add Sparse attention to capturing hip graphs (+13% P90 interactivity on MI355X): #43383
- [AMD][Quark] Serve the Kimi-K3 MXFP4 checkpoint on ROCm (GSM8K 0.942): #40811
- [AMD][DCP] Enable aiter asm ps for kimi k3 prefill: #34438
- [AMD][DCP] add aiter asm for kimi k3 target verify: #34537
- [AMD][Kimi-K3] Use public FlyDSL APIs in KDA decode kernels: #43968
- [AMD][PD] enable mori transfer for PD decode DCP: #37358
- [AMD][Docker] Update MoRI to v1.2.4: #43834
- [AMD] Add an opt-in AITER CK blockscale GEMM for gfx942 block-FP8 linears (+14% for Qwen3-8B-FP8 at concurrency 4): #41071
- [Diffusion][ROCm][Perf]: Set gfx942 AITER FMHA rounding mode to rtz instead of rtna (E2E latency down 5.0% on gfx942): #28650
- [ROCm] Select DSA indexer top-k wave size by arch (wave32 on gfx1250): #41687
- [ROCm] Skip the tilelang act_quant in the DSA indexer on gfx1250: #42747
- [AMD][Diffusion] VAE attention to the math SDPA backend on gfx1250: #42858
- [AMD][Diffusion] Route bf16 fuse_scale_shift to the native fallback on gfx1250: #42859
- [Diffusion] Default Qwen-Image 2.1 to VAE tiling on gfx1151: #41655
- [Diffusion] docs: Add gfx1151 Strix Halo serving guidance: #41670
- [AMD] fix kda decode flydsl import: #40546
- [AMD] AITER MLA DCP decode ASM path (opt-in): #40750
- [AMD] Gate flashinfer and TRT-LLM DSA paths on CUDA: #40911
- [AMD] Keep cos_sin_cache fp32 on HIP for fused QSA indexer kernel: #41282
- [AMD] Resolve QSA packed-varlen decode to aiter on HIP: #41513
- [AMD][DeepSeek-V4.1] Greedy dspark draft/accept under SGLANG_SIMULATE_ACC_LEN: #41981
- [AMD] Drop static input_scale on aiter per-token FP8 path (use dynamic per-token): #42152
- [AMD] Skip the AITER #6042/#5967 patches on gfx1250: #42219
- [AMD] Fix RoPE cache dtype and diffusion CI failures: #42494
- [AMD] Let AITER unified verify run without host seq_lens: #42635
- [AMD] Keep MiniMax decode-score tile at 128 tokens off gfx950: #42772
- [AMD] Patch aiter's group32 MXFP8 GEMM masked-scale fix: #43319
- [ROCm][Spec] Route NGRAM tree reconstruction to the JIT kernel on HIP: #43328
- [AMD] Fix Flux2 kv-gather abort on non-contiguous K/V: #43793
NPU / Ascend
- [NPU] Add decode context parallel support for dsa models: #37787
- [NPU] Quote the arm64 Triton-ascend wheel URL in npu.Dockerfile: #43035
- fix: use dp_size instead of attn_dp_size for msprobe rank id: #43041
- [Fix] Adapt NPU Qwen-VL processor patch to transformers 5.19 resize() signature change: #43309
- [NPU] Bump SGLANG_KERNEL_NPU_TAG to 2026.9.0.post9: #43340
CPU / Intel / XPU
- [XPU] Support decode context parallelism (DCP) on Intel XPU: #34355
- [XPU] Add dynamic_expert_bias and track_state support: #39618
- [XPU] Misc changes to support XPU: #39817
- [XPU] Enable fused QK-norm + RoPE for Qwen3-MoE (60% less device time in the q/k-norm + RoPE stage on 4x Arc Pro B60): #40190
- [XPU] Enable ring attention on XPUAttentionBackend: #41406
- [XPU][Diffusion] Plain-torch scale-shift for inductor: #41885
- [Diffusion][XPU] FLUX.2: enable the bit-exact Triton SwiGLU on XPU: #41940
- [XPU] Upgrade XPU backend to PyTorch 2.14: #42003
MUSA / Moore Threads
- Fix MUSA detection under torch.compile fullgraph: #41065
- [MUSA] Fix fused MoE GEMV registration and torchada pin: #41444
- [Fix] Preserve available host memory for integrated MUSA devices: #43398
Security
- Harden OpenAI upload filename handling: #17270
- fix(security): pin multimodal-gen ZMQ scheduler ingress to loopback on wildcard --host: #36854
- [Fix] Require admin auth for /load_lora_adapter_from_tensors and /set_trace_level: #41322
Cookbook Updates
- DeepSeek-V4 and DeepSeek-V4.1: MI355X DeepSeek-V4 Pro PD recipes (#42757), refreshed AMD recipes and images, and fixed MoRI IO and UMBP playground options: #42847, #43033, #43962, #42512
- SemiAnalysis InferenceX AgentX: interactive launch commands for GLM-5.2, DeepSeek-V4, DeepSeek-V4.1, Qwen3.5 and Qwen3.8-Flash-Next, refreshed for B300, GB200 and GB300: #42726, #43083, #43799, #43983
- GLM-5.2: GB300 NVFP4 environment variables from the AgentX recipe, now in the agentic HiCache section; MI355X daily image and Top-K v2 for MTP: #42109, #42167, #42278, #42613
- GLM-5.3-Flash: the NVIDIA NVFP4 checkpoint, a measured MI355X MXFP4 agentic recipe, and a playground with B300 as default plus CP8, EP and PD: #41986, #43015, #43802
- Qwen3.5, Qwen3.8 and Qwen3: an MI355X FP8 agentic recipe for Qwen3.5, a GB300 long-context recipe for Qwen3.8, H200 mixed-chunk latency tradeoffs and the default attention backend on B300 for Qwen3: #41849, #43159, #43551, #43554
- Kimi-K3: Ascend NPU PD disaggregation recipes and an updated NPU feature list: #41717, #44052
- K2 Horizon: refreshed checkpoint revisions: #42130
- Clef: a new page with a selectable Clef Flash variant and accuracy results: #43235, #43349
- Wan-Animate-2: a command picker: #43380
- MiniCPM-V 4.7: a new page (runs on the
dev-minicpm-v-4.7image): #43829
Dependencies
- PyTorch 2.13.0 to 2.14.1 on CUDA, with sglang-kernel 0.4.9, sgl-deep-gemm 0.2.1.post1, sgl-deep-ep 0.1.2.post1 and torchcodec 0.17.0 (#38641); sglang-kernel 0.4.8 (#41337) and sgl-deep-gemm 0.2.2 (#42604) along the way
- Transformers 5.12.1 to 5.17.0 with tokenizers 0.22.2 to 0.23.2 (#39012), then 5.19.0: #42784
- FlashInfer 0.6.18 to 0.7.0.post1 with nvidia-cutlass-dsl 4.8.0 and quack-kernels 0.6.5 (#40709), then 0.7.1: #43494
- fastokens >= 0.3.2, now part of the
allextra: #43104 - New optional extra
sglang[instanttensor](instanttensor >= 0.1.9): #40453 - Python 3.11+ is required (#43021); the CUDA image uses uv-managed Python 3.12: #42612
- Mooncake 0.3.13 to 0.3.13.post1: #41678
- Fix
pip installby leaving themultimodal_gen/.claudesymlink out of package data: #41703 - XPU: PyTorch 2.14.0+xpu, torchvision 0.29.0, sglang-kernel-xpu 0.3.0 to 0.4.0: #42003
- ROCm: AITER pin for gfx942 / gfx950 moves to
e7d2453f(#42023) and MoRI to v1.2.4: #43834 - NPU: sgl-kernel-npu 2026.9.0.post9: #43340
- MUSA: torchada >= 0.1.90: #41444
Breaking Changes & Upgrade Notes
- SGLang requires Python 3.11 or newer. On Python 3.10,
pip install sglangdoes not fail; it resolves to the last release that supported 3.10. Move to Python 3.11+ (for example off the Ubuntu 22.04 system Python) to get this and later releases: #43021 --enable-dp-attentionis deprecated; use--attn-dp-size N.--dp-sizenow counts model replicas only. The old--dp-size N --enable-dp-attentionstill resolves to--attn-dp-size Nwith a warning; combining replicas with attention DP, or passing both spellings, is rejected at startup: #41818. Under DeepSeek context parallelism,--dp-size Nstill means attention DP but now warns and will be rejected after 2026-12-31: #42348--disaggregation-decode-host-receive-thresholdflips its encoding.1(the new default) disables host receive and0now means always receive on host. Configurations that passed0to disable it must pass1: #43407ChunkCacheis removed. With--disable-radix-cache, full-attention, hybrid-SWA and Mamba models runUnifiedRadixCachein disabled mode and pure-SWA models runPureSWARadixCache(#43198, #43199, #43200, #42354, #43201). Streaming sessions now start only onUnifiedRadixCache; pure-SWA, FlexKV and plugin tree caches fail at startup with a "not verified" error (#42295), and beam search rejects Mamba models with--disable-radix-cachetoo (#42354)- The
tc_piecewiseprefill CUDA graph backend is removed. Passing--cuda-graph-backend-prefill tc_piecewiseor atc_compilersetting now raises an error; CUDA keepsbreakableas the default prefill graph backend, and other platforms default to no prefill graph: #42287 - Renamed and retired environment variables.
SGLANG_AITER_MLA_DCP_DECODE_BACKENDbecomesSGLANG_AITER_MLA_DECODE_BACKENDand its default changes fromgluontoasm(#34537);SGLANG_OPT_SWA_RELEASE_LEAF_LOCK_AFTER_WINDOWbecomesSGLANG_OPT_RELEASE_PREFILL_SWA(#42657); the old names keep working with a deprecation warning.SGLANG_DSPARK_FAST_SAMPLING,SGLANG_DSPARK_FOLDED_SAMPLINGandSGLANG_DSPARK_OPT_FUSED_GREEDY_MARKOVare deprecated and no longer do anything (#43622, #43668), andSGLANG_TRITON_PREFILL_TRUNCATION_ALIGN_SIZEis deprecated: #43444 - OpenAI chat responses changed in three small ways.
tool_choice: "none"no longer shows tool schemas to DeepSeek models (#43006), an absenttool_callsfield is omitted instead of returned asnull(#43027), and a stream keeps onecreatedtimestamp for its whole lifetime: #43026 --enable-layernorm-spis rejected with pipeline parallelism. It failed on the first prefill before: #43458- For model and plugin authors:
ModelRunner.tp_groupis removed; readget_parallel().tp_groupin the scope the code runs in (#41817). Unread placement attributes such asmodel.tp_sizeare gone; readget_parallel()(#41812). A model that declares stage boundaries and runs under pipeline parallelism must pass its shared stage declaration (stage_facts) tomake_layers/make_pp_layers: #43460
New Contributors
- @TwinklerG made their first contribution in https://github.com/sgl-project/sglang/pull/40515
- @googs1025 made their first contribution in https://github.com/sgl-project/sglang/pull/41444
- @ajit283 made their first contribution in #35330
- @anthonyhe-bot made their first contribution in #41486
- @Manfredss made their first contribution in https://github.com/sgl-project/sglang/pull/41314
- @sohom-cs made their first contribution in #41380
- @shafeeqibraheem made their first contribution in #31808
- @stanleys12 made their first contribution in https://github.com/sgl-project/sglang/pull/41322
- @yunzshi made their first contribution in https://github.com/sgl-project/sglang/pull/39788
- @rodamani made their first contribution in #41736
- @shyeh25 made their first contribution in #39388
- @Jin-Chuan made their first contribution in #42002
- @ThomasNing made their first contribution in https://github.com/sgl-project/sglang/pull/41488
- @Tokha233 made their first contribution in https://github.com/sgl-project/sglang/pull/42255
- @livingshade made their first contribution in #37077
- @ekrim made their first contribution in #40703
- @trebladev made their first contribution in https://github.com/sgl-project/sglang/pull/41671
- @JarJarBeatyourattitude made their first contribution in #39862
- @JiaFengYu made their first contribution in #40785
- @cyhdmjzzy made their first contribution in #37261
- @aiyueqi made their first contribution in https://github.com/sgl-project/sglang/pull/40095
- @gpulost made their first contribution in #39523
- @chengshiwen made their first contribution in #22813
- @debanshd made their first contribution in https://github.com/sgl-project/sglang/pull/40158
- @reger-men made their first contribution in #33883
- @yunyang1999 made their first contribution in #32779
- @wildoranges made their first contribution in https://github.com/sgl-project/sglang/pull/36854
- @MaxMiksa made their first contribution in https://github.com/sgl-project/sglang/pull/17270
- @tiankongdeguiji made their first contribution in #42467
- @LiRunGuo made their first contribution in #40671
- @jetd1 made their first contribution in #40888
- @vvvdwbvvv made their first contribution in https://github.com/sgl-project/sglang/pull/41670
- @JayZenith made their first contribution in #19530
- @Seas0 made their first contribution in https://github.com/sgl-project/sglang/pull/41529
- @Zerohertz made their first contribution in #15215
- @TyGu888 made their first contribution in https://github.com/sgl-project/sglang/pull/38028
- @fusheng-ji made their first contribution in https://github.com/sgl-project/sglang/pull/42437
- @pingqiu made their first contribution in #42399
- @adenzhou1350 made their first contribution in https://github.com/sgl-project/sglang/pull/40765
- @Chillee made their first contribution in #42927
- @kunchengit made their first contribution in #31773
- @xuantengh made their first contribution in #39199
- @areneau-cf made their first contribution in #42721
- @murthy-l-habana made their first contribution in #36470
- @cr-gao made their first contribution in https://github.com/sgl-project/sglang/pull/41492
- @R00Kie-Liu made their first contribution in #31768
- @bookinstock made their first contribution in https://github.com/sgl-project/sglang/pull/38671
- @Raymondlol made their first contribution in #34978
- @frgossen made their first contribution in #35755
- @YitaoYuan made their first contribution in #40453
- @colleen-tml made their first contribution in #42807
- @LoCinder made their first contribution in #38217
- @SunskyXH made their first contribution in #43332
- @amd-nprotaso made their first contribution in https://github.com/sgl-project/sglang/pull/43383
- @Phoenix3334 made their first contribution in #35721
- @wanghemeng made their first contribution in #42212
- @ChengYM5685 made their first contribution in https://github.com/sgl-project/sglang/pull/40666
- @C1ves made their first contribution in #41911
- @ix-48 made their first contribution in #43854
- @jyizheng made their first contribution in #43734
- @apex-mochen made their first contribution in #42077
- @Vegetog made their first contribution in #43796
- @BingooYang made their first contribution in #39771
- @michal-shalev made their first contribution in #43723
- @yyh-sjtu made their first contribution in #42127
- @LittleNyima made their first contribution in #35839
- @kailash109 made their first contribution in #41951
- @andrewbriand made their first contribution in https://github.com/sgl-project/sglang/pull/44137
- @chuanrui made their first contribution in #39730
Full Changelog: v0.5.21...v0.5.22
Highlights
779 PRs from 227 contributors.
New models
| Model | Type | Cookbook |
|---|---|---|
| DeepSeek-V4.1 Flash | LLM / VLM | link |
| GigaChat 3.5 | LLM / VLM | link |
| IQuest-Q1 | LLM / VLM | link |
| MiMo-V2.6 / MiMo-V2.6-Pro | LLM / VLM | link |
| Ling-3.0-flash-VL | LLM / VLM | link |
| DiffusionGemma | Diffusion | link |
| Qwen-Image 2.1 | Diffusion | link |
| Anima Base v1.0 | Diffusion | link |
| Ming-Image 0.1 Design / Design-Layer | Diffusion | link |
| FLUX 3 Action | Diffusion | link |
Key features
- PD instances can switch between prefill and decode on the fly, no restart needed (#28403).
- The prefix cache now runs on a Rust core by default (#39627).
- DeepSeek-V4.1 gets 22% faster first token on long prompts (#40352).
- Kimi K3 gets 20.6% higher prefill throughput in PD serving (#40045).
- GLM-5.3-Flash now runs on AMD MI355X, with FP8 / MXFP4 MoE and MTP speculative decoding (#38546, #39341, #39778).
- More accurate results under PP, DP attention, and CP, with layer communication now handled by SGLang (guide, #41557).
- New Decisions API (
/v1/decisions) turns an LLM / VLM into a low-latency classifier and scorer (docs, #41208). - Score API (
/v1/score) scores all candidates in one request (#38965, #41188). - Run MiniMax-H3 with SGLang Diffusion inside ComfyUI (#35990).
To upgrade
uv pip install --prerelease=allow sglang==0.5.21
| Platform | Docker |
|---|---|
| NVIDIA (CUDA 13) | docker pull lmsysorg/sglang:v0.5.21 |
| AMD MI35x | docker pull lmsysorg/sglang:v0.5.21-rocm10-mi35x |
| AMD MI30x | docker pull lmsysorg/sglang:v0.5.21-rocm10-mi30x |
| Intel GPU | docker pull lmsysorg/sglang:v0.5.21-xpu |
| Intel CPU | docker pull lmsysorg/sglang:v0.5.21-xeon |
Full Release Notes
Speculative Decoding
- Pipeline parallelism x speculative decoding (EAGLE/MTP) compatibility: #30775
- Support XQA backend for SpecDec verify: #32269
- [Spec] Windowed draft-decode attention for built-in EAGLE / MTP drafts: #32673
- Avoid materializing GDN QKV tensors during target verification: #33778
- [Spec] Fix CDF boundary handling in
TreeSpeculativeSamplingTargetOnly: #35798 - [Spec] Add LiLiCorr: a candidate-lattice reranker for DFlash drafts: #37462
- Add out-of-tree DFlash extension points: #38740
- [Spec] Add explicit prefill shared-read capability for plugins: #39502
- [Fix] Don't write conv state from the fused KDA verify kernel: #39524
- [Spec] Model-agnostic last-stage draft embedding under pipeline parallelism: #39643
- Use runtime token widths for Triton speculative verification: #39859
- avoid host sync in DSpark prefill slot expansion: #40111
- [Experimental] Preserve speculative decoding during prefill across DP ranks: #40118
- [Spec][PP] Launch extend microbatches before the spec output exchange: #40499
- [KDA] Enable ReplaySSM for GLM-5.3 Flash: #40517
- [LFM2-VL] Add DSpark speculative decoding (1.66x to 2.56x speedup at batch 1): #40651
- [Spec] Support DFLASH for Kimi K3: #40794
- [Fix] Reduce Nemotron MTP attention outputs once: #40800
- [Fix] Capture complete Nemotron auxiliary hidden states: #40801
- [Fix] Skip the DCP target-verify MLA kernel during FlashInfer autotune: #41138
- Fix mixed chunk prefill with DP speculative coordination: #41179
- [Fix] Plan NextN / MTP draft layers as one-layer models and fix the Bailing V2 NextN draft: #41194
Piecewise & Breakable CUDA Graph
- [Fix] Preserve model runner contracts in prefill CUDA graphs: #35452
- [DSV4][BCG] Optimize the heavy memory use of C4 Indexer when BCG is enabled (capture memory down 58% to 18 GB): #36534
- [Fix] Don't free the multi-CTAs KV counter the decode graphs captured: #39175
- [CPU] Avoid prefill CP predicates during decode graph capture: #39690
- [Runtime] Add decode CUDA graph hooks for eager logits processing: #40222
- [Kimi K3] Fix CUDA graph stream explosion: #40640
- [DSpark] Fix draft CUDA graph stream explosion: #40658
- [Fix] Give the full prefill CUDA graph replay view the captured bucket's input_ids: #40851
- [DSA] Fix the pooled-indexer breakable prefill bridge under DP attention: #41311
Attention Backends
- Allow attention layers to opt out of the prefill wrapper: #40683
- [Fix] Use cached prefix lengths for FlashInfer full-attention ragged prefill: #40796
MoE & Expert Parallelism
- [3/N] elastic-ep: Recapture decode CUDA graphs after scale-up: #33723
- Fix int4 MoE tuner config filename: #35260
- [MegaMoE] Wire Qwen MoE blocks to DeepGEMM MegaMoE (MXFP4 and NVFP4 experts): #38080
- [Feature] Support BF16 and batch-invariant inference with DeepEP v2: #38160
- [Perf] Optimize w4a8 MoE for glm5.2 on H200: #38220
- [Fix] Guard FlashInfer CUTLASS MoE against 0-token inputs: #38780
- [Kernel] Add H20 block-FP8 MoE configs for GLM-5.3-Flash EP4/EP8: #38913
- Accept MXFP8 dispatch in FlashInfer A2A TRT-LLM MoE: #39613
- [MoE] Honor swiglu_limit clamped activation in flashinfer_cutlass runner: #39939
- Support MXFP8 and deferred route weighting in DeepEP v2: #40030
- [MoE] Disable FlashInfer fused finalize by default for numerical accuracy: #40105
- [Fix] Fix top-1 MoE routing with non-unit scaling: #40187
- [DeepEP v2] support GLM-5.3-Flash (Glm5NextForConditionalGeneration): #40466
- [Fix] Decide the MoE padded-row bound from the layer scatter mode: #40672
- [DeepEP v2] Let a model package supply its per-rank prefill dispatch bound: #41201
Quantization
- [MXFP8-KV] Skip writes to the reserved CUDA-graph padding slot: #35351
- feat(kv-cache): support SM100 NVFP4 GenMHA and speculative decoding (1.355x faster decode attention kernel at 1M context): #36340
- [Quant] ModelOpt mixed precision: dispatch block-FP8 MoE experts and derive the block size: #38726
- fix(modelopt): dispatch NVFP4 MoE on the cached backend, not the live global: #38932
- [Quant] Serve 32-wide-K ue8m0 block-FP8 linears through the FlashInfer MXFP8 GEMMs: #40039
- [ModelOpt][PP] Keep BF16 shared experts out of the NVFP4 fusion so TP1 pipeline stages can load: #40628
Parallelism & Disaggregation
- [PD] Introduce runtime role switching between prefill and decode: #28403
- disaggregation: Fix FakeKVSender queue accumulation: #28652
- [PD] Keep EAGLE DP graph and token metadata consistent: #32196
- [NVIDIA][comm] Merge EP+MoE-TP post-experts all-reduces into one _TP reduction: #32963
- Add 8-node AllReduce/AllGather and MNVLS algorithm support to MSCCL++: #37442
- [KV-Shard 2/4] Sharded pools: #37615
- [KV-Shard 3/4] Enable Control plane: #38468
- [KV-Shard 3/4] Enable Control Plane B: #39964
- Fix device context during NIXL backend initialization: #38774
- [PD] Do not admit intake-rejected requests to a PD handoff: #38935
- Reduce decode bootstrap latency with request-owned speculative KV (mean TTFT 19.4 to 15.1 s on DeepSeek-V4-Pro 1P1D, MI355X, concurrency 256): #38978
- Fix disagg PP MTP for GLM-5.2: #39378
- [PD] Add optional KV transfer checksums: #39500
- [DCP] Use logical token capacity for PD admission and load reporting: #39731
- Refactor the Cute-DSL AR fusion to support DeepseekV2 archs (GLM-5.3, etc.): #39816
- [PD] Validate Mooncake EFA allocator compatibility: #39973
- [PD] Skip singleton transfer-status all-reduces: #40003
- [PD] Enable optimistic prefill with buffer-only L3 write-through HiCache: #40043
- [Kimi K3][PD] support pp prefill + dcp decode with dspark: #40045
- [PD] Add decode host receive for custom transfer backends: #40238
- [PD] Allow decode radix cache and HiCache L1/L2 with DCP: #40263
- [PD] Enter the custom mem pool once when allocating DCP pack buffers: #40284
- [PD] Bound cached-prefix DCP transfers by pack capacity (cached-prefix throughput about 10.9x higher at concurrency 8 on Kimi Linear, 8x B200): #40376
- [PD] Pack draft KV head slices for DCP transfers: #40500
- [PD] Preserve abort ACKs until in-flight KV transfers drain: #40645
- Fix NIXL transfer of MXFP8 KV block scales: #40792
- [PD] Honor gracefully_exit in disaggregation event loops and keep non-zero-rank launchers alive on SIGTERM: #40793
- [DP Attention] Publish DP buffer sizes from a ForwardBatch: #40858
- [PD] Enable deferred decode-side KV release by default: #41023
- [PD] Add a
nonedecode retraction backup and subclass seams in the PD queues: #41103 - [PD] fix: cache resumed decode-radix requests from root instead of an unlocked re-match: #41261
- [PD] Fan drain abort ACKs out to every decode peer of the room: #41402
- [PD] Defer decode KV release on every transfer failure, not only decode-initiated aborts: #41404
- [Fix] Stop deferring the last layer's FFN all-reduce in five models: #40868
- [Fix] Complete the deferred FFN all-reduce before a pipeline-parallel send: #41079
- [Fix] Complete the deferred FFN all-reduce before deepstack addition and aux hidden-state capture: #41080
- [Fix] Step-3.5: stop dense layers from summing their output twice under DP attention: #41082
- [Fix] Broadcast requests along attention CP before attention TP: #41083
- [Fix] Complete the all-reduce when the flashinfer fused norm declines a batch: #41193
- [Fix] Stop counting a deferred FFN sum more than once: replicated TP1 shared expert, dense reduce_scatterv: #41195
- [Fix] Keep one copy of CP-replicated rows in the DP gather: #41422
- [Fix] Gather a dense FFN's input across attention DP and CP in one DP sum: #41423
- [Fix] Run MoE layers under attention DP and GQA prefill CP on the declared DP × CP gather: #41432
- [Fix] Falcon-H1: count the Mamba mixer's output once under tensor parallelism: #41433
- [Fix] LongCat-Flash under attention DP: branch and merge the dense FFNs through the communicators: #41436
- [Refactor] Group communicator fusion and CP adapters: #41547
- [Refactor] Centralize decoder output access: #41548
- [Refactor] Carry residual state across stage boundaries: #41549
- [Refactor] Capture auxiliary states at residual reads: #41550
- [Refactor] Select reduction fusion at the consumer: #41551
- [Refactor] Construct independent decoder stage boundaries: #41552
- [Refactor] Migrate specialized decoder and overlap boundaries: #41553
- [Refactor] Retire the layer facade and simplify boundary internals: #41554
- [Refactor] Rename the module to layer_boundary: #41555
- [Refactor] Group layer boundary unit tests: #41556
- [Refactor] Document layer boundary contracts and integration: #41557
Scheduler & Runtime
- feat: support custom OTLP trace service name: #35802
- [RL] Release the weight-checker snapshot once compare passes: #37284
- [Runtime] Let out-of-tree platforms provide full graph backends: #37969
- Fix: post-load staging regression breaks offload meta/sharded_gpu modes: #38779
- feat: add kv hint envelope to request transport: #38891
- [Observability] Fix negative queue_time for retracted requests: #39312
- Fix first-token metadata and reused attention-layer indexing: #39328
- perf(engine): avoid timed waits for Engine responses (+37.7% output throughput in a Dynamo TCP push benchmark at the default batch_notify_size): #39486
- [Misc] Fix tool-call index, graph padded-row count, and prefill-graph input_embeds refresh: #39574
- [Misc] Merge FlashInfer autotune caches across spec workers, pad MXFP4 TP shards, drop dead ngram attrs: #39678
- [Observability] Expose python/rust frontend identity in
/server_info: #39993 - [Metrics] Propagate idle gaps across all scheduler loops: #40004
- [Scheduler] Count complete prefill bursts and their tokens: #40006
- [Scheduler] Add shortest-prefill-first scheduling: #40024
- [Profiler] Ignore PREBUILT batches in profile-by-stage: #40098
- Enable optimistic prefill for Mamba radix-cache models: #40184
- [Perf] Fork-safe import: no CUDA context at import time, lighter argument parsing: #40201
- [Feature] Support --tokenizer-worker-num > 1 in the offline Engine API: #40260
- Clean up startup logging and streamline log audits: #40526
- [Fix] Run KV canary hooks for context-parallel prefill: #40642
- [RL] Fix Kimi K3 expert-count lookup for routed-expert capture: #40700
- [RL] Add RL weight-update sessions and support updating spec draft runners: #40777
- [RL] Keep pause_generation and weight updates from deadlocking each other: #40779
- [Misc] Remove deprecated endpoints, env vars and aliases past two releases: #40795
- [Metrics] Log forward and forward+idle occupancy over total wall time: #40802
- [RL] Keep DSA cuda-graph state and the graph pool intact across TMS pause/resume: #40804
- [Fix] Recover from stale torch extension locks in every
cpp_extensionloader: #40989 - [Perf] Lazy-load built-in model definitions and nixl_ep at startup: #41061
- [Model Loader] Stop checkpoint prefetch after iterator completion: #41588
- [Benchmark] Limit warmup concurrency in serving benchmark: #39398
- Support LongBench v2 in one-batch server benchmarks: #39874
- [Benchmark] Take each request's prompt length from the server: #39889
- [Benchmark] Add agentic rollout simulator and offline explorer: #40034
- [Benchmark] Optionally clear HiCache storage between cases: #40659
Sampling
- fix(sampling): validate sampling_seed is an int within int64 range: #28960
- perf(sampling): avoid GPU syncs when applying custom logit processors: #39234
- Use pinned memory for asynchronous sampling metadata transfers: #39777
- [Logprob] Borrow graph-pool memory for input logprob logits construction: #40007
- [Logprob] Serve input-logprob temporaries from CUDA-graph-pool dead space: #40038
- [Sampling] Add selected/support sampling logprob modes: #40932
- [Sampling] Stream sampling masks as per-request arrays: #40986
- [PD] Keep the sampling mask of a replayed rebootstrap token: #41235
HiCache & Radix Cache
- [Radix Cache] Sync Rust TreeCore and make it the default: #39627
- Add Agentic-Aware Tail-Optimized LRU eviction to the unified radix cache (opt-in; p99 ITL down 43.9% at concurrency 32 on DeepSeek-V4-Pro FP4, 4x B300, agentic trace): #34012
- Feat: Add TensorCast storage as a new HiCache backend: #27265
- [HiCache] fix: resolve Mooncake local_hostname per node for runtime attach: #29668
- [Fix] fix(hicache): wait for decode offload before retraction: #30899
- [HiSparse] Add MHA hisparse support for MiniMax M3: #31446
- [Mooncake] Fix silent SSD offload corruption when TP/PP ranks share ssd_offload_path: #31926
- [Scheduler] Align
RadixCacheno-insert cleanup withkv_len_to_handle: #35204 - [PP + HiCache] Add PP Prefetch Tickets for eager cross-stage storage prefetch: #36700
- Use a shared byte budget for unified hybrid-SWA memory: #36729
- Gate mamba extra-buffer predicates on uses_mamba_radix_cache: #37474
- [Unified Memory] Hierarchical cache for every unified pool shape: #37507
- [HiCache] Fix sparse hybrid transfer layer IDs: #37870
- [Unified Memory] fix: preserve FP8 dtype in unified MHA pool: #38133
- [HiCache] Yield idle scheduler so storage workers can drain: #38504
- [LMCache] Support lmcache unified radix cache: #38652
- [Unified Cache] Dedup replicated MLA/DSA KV in the UMBP direct linker: #38778
- [Perf] Fuse SWA page lookup and mapping clear: #38948
- [HiCache][Perf] fix: batch HiCache D2H submits per step for hybrid pools: #39050
- [HiCache] Back up MXFP8 KV scales in the host pool: #39089
- [Mamba] Fix checkpoint depth for prefixes that end off the radix page: #39115
- [HiCache] Label radix-cache metrics per rank and split the "shrunk" prefetch reason: #39280
- bugfix:fix unifiedcache c128 radix cache management: #39426
- Support unified memory page-envelope transfers in PD: #39477
- Support unified memory decode host pools: #39478
- [Unified Cache][9/N] add opt-in MLA load deduplication for Mooncake Linker (TTFT 3.85 to 2.23 s with a 256K cached prefix, GLM-5.2 on 8x H20): #39565
- [HiCache] Forward prefix metadata to v2 storage calls: #39567
- [HiCache] Keep hybrid transfer layer maps stage-local under PP: #39699
- [HiCache] Add the page-unified KV load-back JIT kernel: #39726
- [2/N][Kernel] Fuse padding-preserving HiSparse slot translation: #39837
- [Unified Tree] fix: exempt host-locked aux nodes from the sanity_check host-LRU check: #39980
- [HiCache] Read the in-flight buffer backup's node id from its snapshot in sanity_check: #40013
- [HiCache] Stop arming a prefetch retry for a too-short storage span: #40042
- [MemCache] Release up to
owned_kv_lenon radix cache insert: #40075 - [HiCache] Auto-size the host pool to fit available host memory: #40135
- Preallocate HiCache MHA staging before post-capture KV sizing: #40256
- Fix prefetch attempt cleanup on abort: #40262
- [HiCache] TMA-staged host<->device KV transfer kernel (sm_90+): #40278
- [HiCache] Size MHA host pools from device row width: #40304
- [Fix] Missing SWA eviction during decode preallocation: #40309
- [HiCache] fix: bound the controller reset join so a stalled storage thread cannot hang the scheduler: #40312
- Remove swa and mamba radix cache: #40313
- [Fix] Forward SWA prealloc reclaim through the DSV4 HiSparse allocator: #40354
- [HiCache] Give trailing sidecar storage transfers a contiguous prefix_keys chain: #40456
- [HiCache] Make host reclamation independent of transfer order: #40512
- [HiCache] Demote internal-node mamba states on write_back eviction: #40680
- [HiCache] Demote internal-node SWA KV to host on write_back eviction instead of dropping it: #40712
- [MemCache] Remove the experimental C++ radix tree: #40775
- [MemCache] Clean up SWA/Mamba radix cache leftovers and drop SGLANG_ENABLE_UNIFIED_RADIX_TREE: #40780
- [HiCache] Remove the unused HiRadixCache: #40787
- [MemCache] Free the rows below the SWA evict floor on all-SWA request release: #40798
- [HiCache] Batch buffer-only KV backups within each flush: #40960
- [Fix] Derive per-runner hybrid SWA layer ids on ModelLayerInfo instead of mutating ModelConfig: #40983
- [HiCache] fix: Drain pending backups before internal Mamba write-back: #41092
- [Unified Memory] Fix Inkling conv-checkpoint track ids written to virtual slot numbers: #41144
- Let predicate-registered linear-attention models carry the mamba radix-cache leaves: #41165
- [Unified Memory] Honor move gates in float relocation and size auto HiCache from host capacity: #41248
- [MemCache] Unify component eviction cursors and lock receipts: #41276
- [MemCache] Never free the protected prefix on request release: #41312
- Make sliding-window caching and speculative batch padding extensible: #41325
- [MemCache] Fix LMCache component cursors and per-cache backend selection: #41328
LoRA
- [LoRA] Support DP attention in LoRA backends: #36389
- [LoRA] Size dense row/column-parallel LoRA buffers from the base linear's real shard: #39379
Multimodal
- fix(multimodal): return 400 for corrupt image inputs: #28131
- [Multimodal] Avoid CUDA placement on non-CUDA platforms: #38750
- Fix Mistral3 retaining every vision-tower layer to read one: #39185
- perf(multimodal): offload CPU feature hashing with bounded admission: #39539
- Carry deferred attention operands and reuse multimodal shared memory: #39870
- [MM] Skip VMM error gathers for text-only requests: #40005
- [MM] Copy placeholder ids to CUDA asynchronously: #40010
- [MM] Keep scheduler padding in packed token arrays: #40357
- [Fix] Raise on undelivered embeddings in
send_with_url, fix broken tests: #40502 - Fix multimodal feature offload races: #40621
- [VLM] Introduce FA4 into ViT for SM100/SM103: #41344
Model Support & Optimizations
- [Feature] Gigachat 3.5 support: #29189
- [dLLM] Support DiffusionGemma serving: #34061
- Add Ling-3.0-flash-VL model support: #38526
- dsv4.1: remaining model and runtime integration: #38798
- dsv4.1: standalone kernels and Python wrappers: #39646
- dsv4.1: Top-k kernels and candidate selection: #39648
- dsv4.1: compression, KV I/O, and metadata kernels: #39652
- dsv4.1: communication kernels and wrappers: #39653
- dsv4.1: RoPE and FP4 packing kernels: #39656
- dsv4.1: Hopper FP8 matmul kernels and tuning: #39657
- dsv4.1: mHC computation and compensated projections: #39664
- dsv4.1: chat encoding and tool parsing: #39665
- dsv4.1: Engram module and request history support: #39666
- dsv4.1: vision tower and image preprocessing: #39668
- dsv4.1: candidate indexer library: #39671
- dsv4.1: Rust extension modules for image preprocessing, KV pool names, and PD bootstrap: #39677
- [DSv4.1] Score prefill consumer index layers on candidate blocks with DeepGEMM: #40352
- [Feature] Xiaomi MiMo-V2.6/MiMo-V2.6-Pro day0 support: #40448
- [Model] Add IQuest Q1 support and MTP draft: #41590
- fix(gemma4): set lm_head_is_tied for Gemma4UnifiedForConditionalGeneration: #35809
- [Fix] Preserve YaRN scaling when extending rotary caches: #38786
- [Kimi-K3] O(1) expert weight lookup in load_weights (expert-name matching 21.45 to 0.13 s at load): #38805
- [Model] Serve DeepSeek-OCR-2 with its official 768px local-crop geometry: #38996
- Fix GLM-OCR MTP multimodal embeddings and positions: #39088
- [DSV4] Chunk the indexer MQA logits by query rows under a free-memory budget: #39095
- [Perf] Fuse the glm5_next mHC attn->MLP boundary: #39200
- [DSV4] fix: keep the TileLang JIT cache under SGLANG_CACHE_DIR: #39364
- Fuse GLM-5.3-Flash KDA projections and prefill metadata: #39688
- [GLM-5.3-Flash] Reduce KPool planning synchronization and overlap indexer preparation: #39695
- [Fix] Fix GLM5 mHC PP forward: #39720
- [DSV4] Generalize attention metadata, sparse prefill, and KV pool over compress ratios: #39921
- [Qwen3.8 Next] Fuse Qwen PLE gate and convolution preparation for target verify: #40041
- Support GLM-5.3-Flash hybrid attention CPU offload and PD index mapping: #40310
- [DSV4] fix: size the C4 state ring by the page it is addressed by: #40337
- [Fix] Keep mHC context out of non-V4 compiled MoE forwards: #40353
- [Qwen3.8-Next] Pipeline-parallel serving and PD-prefill MTP for Qwen4-Exp: #40501
- Fix GLM-5.3 forget-gate shape for nvCUTEDSL verify: #40607
- [Fix] Avoid duplicate residual in LongCat MoE shortcut: #40799
- [DSA] Chunk the kpool indexer MQA logits by query rows under a free-memory budget: #40854
- [DSV4] Budget the ratio-2 pair state pool in DSV4PoolConfigurator: #41048
- [DSV4] Size compressed pools from one per-ratio table in DSV4PoolConfigurator: #41049
- [Fix] Add name mapping in load_weights of nvidia/LocateAnything-3B: #41059
- [DSV4] Fix TRTLLM uniform FP8 KV memory budgeting: #41090
- [DSV4] Account for FlashMLA physical KV page padding in memory budgets: #41091
- [Kimi-K3] Merge fused_qkvg_proj into the loader-seeded packed_modules_mapping: #41164
- [Qwen3.8 Next] Fuse small CUDA graph input buffer copies: #41166
Kernel Library
- [Kernel] Share the warp vectorized copy and enforce its alignment: #36176
- fix(kda_prefill): fence shared writes before async proxy reads: #39124
- [Kernel] Coalesce the KDA CuTe DSL decode state transpose: ~3x faster, bit-identical: #39680
- [Kernel] Move CUDA and ROCm speculative kernels to JIT: #40033
- Fix TopK v2 fallback when 16-block cluster capacity is zero: #40163
- [Kernel] Fuse hc_combine_norm for mid-size verify batches (9-96 rows): #40208
- [KDA] Fix missing beta sigmoid in PTX prefill: #40685
- [JIT] Add an occupancy-preserving L1 carveout preference: #40767
- [Perf] Mamba2 selective_state_update up to 2x faster on B200 via 8x1 launch config for dstate 128 (+7.5% Nemotron-3-Super serving): #41223
Configuration System
- Add registration for external model configurations: #39452
OpenAI-Compatible API
- [Fix] Guard conditional top-logprob keys in the completions echo path: #34776
- [Fix] Prevalidate JSON Schema support per grammar backend: #37839
- fix(openai): recover logprobs token bytes from token_id (UTF-8 fragments): #38604
- [Score API] Setwise Scoring Support: #38965
- feat: use XGrammar V4.1 DSML parameter constraints: #39026
- fix(function_call): buffer complete DeepSeek DSML invokes: #39632
- Fix corrupted chat prompts on mistral_common tokenizers (tool_choice auto never fires): #39773
- [Fix] Keep Inkling automatic tool grammar active across the response: #40468
- fix(grpc): expose native response timeout as a server argument: #40644
- [Feature] Add per-item candidate token scoring and calibration: #40826
- Add MiniMax arch fallback to auto parser resolution: #40930
- [Score API] Setwise scoring: CausalLM support (batched + --enable-mis): #41188
- [Feature] Add a System One compatible /v1/systemone route (also ships the
/v1/decisionsroute it builds on): #41208 - Fix chat template cache key order: #41517
Rust Server
The experimental Rust router and frontend (experimental/sgl-router, the Rust renderer) keep moving toward parity with the Python server.
- [Rust Renderer] Standalone preprocessing: #36718
- [SGL Router] Render chat prompts with dynamo-render: #38983
- [Router] Fleet-wide sampling contract 1/3: the config surface: #39000
- [Router] Fleet-wide sampling contract 2/3: enforce and inject per request: #39001
- [Router] Fleet-wide sampling contract 3/3: splice injection without re-serializing: #39002
- [Router] Resolve a wire protocol per worker at registration: #39004
- [Router] Speak cleartext h2c on both edges: serve it inbound, forward it outbound: #39006
- [Router] Count open HTTP exchanges until their response body finishes: #39014
- [Router] Add the shutdown-drain configuration surface: #39015
- [Router] Drain readiness before SIGTERM shutdown so k8s deregisters the pod first: #39016
- [Router] Real-GPU e2e coverage for storage-tier-aware cache routing (3/4): #39110
- [Router] Add a KV storage-tier coverage row to the Grafana dashboard (4/4): #39111
- [Router] Improve SGLang chat render parity: #39133
- [Router] Shard the cache-aware KV tree by chain root: #39167
- [Router] Add --worker-queue-limit: stop sending cache-affinity traffic to a queueing worker: #39168
- [Router] Pin to the prefix owner when the whole fleet is queueing (--saturation-queue-floor): #39169
- [Router] Sample k random candidates for the min-load fallback (--min-load-choices): #39170
- Add external multimodal processors to the Rust frontend: #39329
- [Rust] Extract a transport-neutral frontend core: #39385
- [SGL Router] Forward input_ids only for string content; count tokenize errors only when forwardable: #39458
- [Router] Abort the engine when a client disconnects mid-request: #39461
- [Router] Derive error status from a failure class; preserve the worker's status (1/3): #39463
- [Router] Treat an upstream 503/429 as backpressure, not a breaker fault (2/3): #39464
- [Router] Log every request at one site; derive its outcome from the final status (3/3): #39465
- [SGL Router] Share model-file discovery for chat formatters: #39485
- Rust server unify datapath for mm and generate requests: #39679
- [SGL Router] Add Kimi-K3 rendering with SGLang parity: #40390
- [SGL Router] Bound streaming lifetimes and release guards on idle disconnect: #40391
- Port chat_parsing core: #40477
- [SGL Router] Match DeepSeek V4 rendering to SGLang: #40530
- [SGL Router] Add SGLang-compatible DeepSeek V4.1 Flash rendering: #40532
- [SGL Router] Release cancelled circuit-breaker probes: #40603
- [SGL Router] Fix readiness, IPv6 discovery, logging, and model validation: #40604
- [Router] Give the cache-aware tree a snapshot surface (1/13): #40687
- [Router] Serve the cache-aware tree at /internal/kv_snapshot (2/13): #40688
- [Router] Name a replica's siblings with --kv-peer-selector (3/13): #40689
- [Rust Renderer] decouple renderer sampling from protocols: #40747
- [SGL Router] Launch reorg routing with existing policy options: #40766
- [SGL Router] Track input_ids forwarding outcomes per chat request: #41185
- [SGL Router] Take SGLang's render defaults: --default-chat-template-kwargs and thinking/effort envs: #41221
- [SGL Router] Scope input_ids forwarding by renderer: all text chats for DeepSeek-V4: #41226
- [Rust Frontend] Decode input_ids without untagged buffering (input_ids deserialization about 45% faster): #41246
- [SGL Router] Book the input_ids forwarding outcome only for built bodies: #41342
Simulator
- [Simulator][Compatibility] Adapt to latest KV cache pool interfaces: #40418
SGLang-Diffusion
- [Diffusion] model: support qwen-image-2.1: #39983
- [Diffusion] Add native Anima Base v1.0 support: #41011
- [Diffusion] Support FLUX 3 Action robot policies: #41066
- [Diffusion] Add native Ming-Image Design and Design-Layer support: #41067
- [Diffusion] add /metrics support: #19084
- [Diffusion] Preserve per-sample rollout trajectories across multi-output merge: #34416
- [Diffusion] Fix AttributeError in grouped forward_batch by installing the residency manager: #34417
- [Diffusion] Apply latent-ids and packing to caller-provided initial latents: #34418
- [Diffusion] MiniMax-H3 Spectrum skip-step + fused RMSNorm/AdaLN: #35684
- [Diffusion] Add MiniMax-H3 to ComfyUI integrated mode: #35990
- [Diffusion] In-place LoRA merge/unmerge under layerwise offload: #36192
- [Diffusion] feature: out of tree platform support: #37547
- [Diffusion] support multiple task types for pipelines: #38762
- [Diffusion] Enable shared RMSNorm dispatch for SenseNova-U1: #39705
- [Diffusion] Preserve explicit attention backends during autotune: #39882
- [Diffusion] Cache-DiT 1.5.1: DMD Calibrator, SVDQuant DQ, etc.: #40104
- [Diffusion][MiniMax-H3] Add SM120 Sage compute for SubBlock sparse attention: #40116
- [Diffusion] attention: add fp8_fa_sm120 FP8 backend for SM120 GPUs: #40175
- [Diffusion] Fuse lossless SenseNova RoPE for 5% faster H200 inference: #40374
- [Diffusion] Fuse rounded SwiGLU for quantized MiniMax-H3 MLPs (FastH3 FP8 worker E2E 1.9% lower on 2x H200): #40378
- [Diffusion] Fuse LongCat GELU+cat and support Edit-Turbo BCG: #40384
- [Diffusion] Accelerate Cosmos3 Edge on Hopper with lossless fusions: #40386
- [Diffusion] Enable lossless SANA-Video eager conv fusions for 12.6% lower latency: #40388
- [Diffusion] Fuse lossless Wan VAE post-ops for LongLive 2 I2V (worker E2E 5.5 to 5.9% lower on one H200): #40405
- [Diffusion] Fuse lossless LingBot World FP32 normalization: #40425
- [Diffusion] Populate CPU weight stores before host registration: #40439
- [Diffusion] Add bounded exact conditioning cache across native models: #40470
- [Diffusion] Enable lossless Cosmos3 Super T2I QK fusion on Hopper TP2 (worker E2E 7.4 to 8.8% lower on 2x H200 at TP2): #40486
- [Diffusion] Fuse lossless Klein packed QK RMSNorm and RoPE on Hopper: #40490
- [Diffusion] Fuse Joy Image Edit QKV concatenation and avoid QK copies: #40494
- [Diffusion] Support MiniMax-H3 PDD(Parallel Decoding Distillation) inference: #40568
- [Diffusion] Separate a use-scoped layerwise release from release_all: #40590
- [Diffusion] feat: allow a component use retain its layerwise resident set: #40592
- [Diffusion] Add a permanent lifetime for layerwise resident layers: #40599
- [Fix] Keep diffusion encoder TP context bindings consistent: #40646
- [Diffusion] Add opt-in SRT prompt enhancement to image and video APIs: #41095
- [Diffusion] fix: LoRA-wrapped linears crash Qwen-Image and MiniMax-H3 inference: #41272
- [Diffusion] Copy small files into overlay materialized trees instead of linking them: #41298
- [KDA+Kimi K3] Speed up SANA-Video residual gate add on H200: #41305
- [Diffusion] Run an all-valid attention mask on the backend's unmasked kernel (Ideogram4 request latency 3.42 to 2.19 s at TP2 on B200): #41309
Local & Desktop AI
- [SM120] Add optional FlashInfer PCIe-IPC all-reduce for switch-free hosts: #34528
- fix(moe): support Llama4 NVFP4 router input weights on SM120: #35504
- [PP][DeepSeek V4] Overlap communication and optimize SM120 prefill: #38792
- [Fix] Include SM121 in DeepGEMM packed-scale selection: #39482
- [Qwen4-Exp] Build the offloaded PLE table on the meta device so --ple-offload-embedding never materialises it on the accelerator: #39928
- Pin triton_kernels num_warps for MXFP4 MoE below Hopper (6x gpt-oss decode on RTX 4090): #41292
AMD / ROCm
- [AMD] Speed up Wan2.2 DiT FP8 attention per-tensor quantization: #34695
- [AMD] Fix DSV4 FP4 dequant path for AITER on ROCm: #35123
- [ROCm][Diffusion] Enable fused qk norm and rope on ROCm: #35573
- [AMD] Integrate Aiter MegaMoEv2 for DeepSeek-V4: #35619
- [AMD] Skip full-vocab softmax in EAGLE topk==1 draft on ROCm: #35872
- [ROCm][Perf] aiter: page-level KV view for gfx950 fp8 page-64 asm prefill: #36505
- MiniMax-M3: run the sparse prefill main attention through AITER Gluon paged attention: #36546
- MiniMax-M3: allocate the lightning-indexer K cache in fp8 on gfx95: #36549
- MoE: small-batch sorting path with fused mxfp8 quantisation: #36559
- MiniMax-M3: wave64 histogram-select decode top-k, and raise kMaxNumBlocks for CUDA graphs: #36560
- MiniMax-M3: MXFP8 dense-only block convert + aiter MXFP8 MoE on gfx950: #36574
- MiniMax-M3: allow shared-experts fusion on ROCm gfx942 and newer: #36576
- [ROCm] Fix EAGLE spec-decode verify silently sampling greedy on HIP: #37134
- [ROCm] Widen the HiCache JIT copy rounds and enable the K-only host pool: #37152
- [AMD] Preserve deterministic inference when Lean Attention is enabled: #37740
- [AMD][Diffusion] FlyDSL fused norm kernels on wave32 targets (gfx1250): #37751
- [AMD] Fix DeepSeek-R1-MXFP4 accuracy with AITER FP8: #37762
- [AMD][DSV4] Enable hicache on deepseek-v4 fp8 unified attn: #37778
- [ROCm][DSV4] Enable breakable CUDA graph prefill: #37810
- [AMD] Enable GLM DSA prefill top-k to the v2 kernel (+4.9% token throughput at 70K input on GLM-5.2-MXFP4, 4x MI355X): #37889
- [AMD][Spec] Enable GDN ReplaySSM target-verify on ROCm: #38184
- [ROCm] Fuse the MLA q absorb into the RoPE + KV-write kernel on gfx950: #38340
- [AMD] Avoid the FP8 wo_a path when the weight is BF16: #38453
- [AMD][GLM-5.3-Flash Day 0] Route mHC through AITER on gfx950: #38545
- [AMD][GLM-5.3-Flash Day 0] Enable FP8 and Quark MXFP4 MoE on gfx950: #38546
- [AMD][GLM-5.3-Flash Day 0] Enable zero-RoPE TileLang DSA on gfx950: #38547
- [AMD][GLM-5.3-Flash Day 0] Honor fused and per-expert names in quark
exclude: #39317 - [AMD][GLM-5.3-Flash Day 0] Enable zero-RoPE MHA prefill on ROCm: #39338
- [AMD][GLM-5.3-Flash Day 0] Build the fused DSA k-pool top-k JIT kernel on HIP: #39339
- [AMD][GLM-5.3-Flash Day 0] Support non-2048 top-k widths in the DSA page-table transform: #39340
- [AMD][GLM-5.3-Flash Day 0] Enable the k-pool DSA indexer on gfx950: #39341
- [AMD][GLM-5.3-Flash Day 0] Enable speculative decoding (MTP) on ROCm: #39778
- [AMD][GLM-5.3-Flash Day 0] Load the MXFP4 MTP draft layer: #39779
- [AMD] Pad QSA MQA decode Q-heads to 16 for ROCm MFMA: #38875
- [AMD] Add a Triton packed sparse decode path for QSA on ROCm: #38876
- [AMD] Load fused shared experts for Qwen4-Exp and Qwen3.5 MTP: #38878
- [AMD][DSV4] feat: enable DSpark with fp8 unified_kv on gfx950: #38901
- [AMD] Tune Triton sparse MLA on gfx950 and make split-K workspaces graph-safe: #39059
- [ROCm][Fix] Keep quantization for mixed Quark Qwen3.5 MTP checkpoints: #39064
- [AMD][Kimi-K3] Fix deferred KDA gate projection and update DCP cookbook: #39066
- [AMD] GLM-5.2 NextN: cast draft fused MoE to per-channel FP8: #39155
- [AMD] Use exact CU share for gfx950 segment-plan headroom: #39503
- [AMD][Fix] Fix vattn_asm HIP error 709 under CUDA graph capture on ROCm 10: #39513
- [AMD] Fix deferred Kimi-K3 forget gate in fused in-projection: #39525
- [AMD][Fix] Fix DSV4 MTP crash: #39547
- [AMD] Prefer HIP Top-K for GLM-5.x on ROCm (P90 interactivity 96.97 to 106.69 tok/s/user on MI355X at concurrency 8): #39631
- [AMD] Update deepseek-v4 PDI and cache policy setting for agentic workload: #39702
- [AMD] Clamp MORI intranode grid GPUs: #39763
- [ROCm] fix: remove extra bf16 -> fp32 cast in jit grouped topk kernel path: #39775
- [ROCm] feat: enable aiter allreduce fusion for GLM models: #39790
- [AMD][Fix] Fix dsv4 server launch: #39875
- [AMD] Reuse KV gather indices across ASM context prefill layers: #39901
- [AMD] Pack Qwen3.5 GDN input projections on ROCm: #39902
- [AMD][DSV4] Allow moe_a2a_backend='mori' with DSpark + dp attention: #39910
- [AMD] Update ROCm AITER pin to acf8fdf9: #39965
- [AMD] dsv4: pick kv_splits per index stream, not by occupancy alone: #39968
- [AMD] Use Triton softmax routing for Qwen3.5 on gfx950: #39986
- [AMD] Tune Qwen3.5 TP4 GDN recurrent launch on gfx950: #39987
- [AMD][DSV4] fix: drop shadowing local get_exec import that breaks model startup on ROCm: #40186
- [AMD] Small-M MXFP4 fused-MoE kernel for gfx950 (Qwen): #40204
- [AMD] Drop the redundant scale zero-fill before AITER per-tensor FP8 quant: #40557
- [ROCm][DSA] Enable AITER fused FP8 indexer writer: #40710
- [AMD] Critical fix enabling Qwen3.8 FP8: restore dropped fused shared-expert weights: #40754
- [AMD] Add diffusion (Wan2.2) extras to gfx1151 Docker image: #40844
- [AMD][DSV4] fp8 unified_kv decode: wave-aware split count past 40 tokens: #40878
- [AMD] Drop the unreachable vLLM fallback from ROCm FP8 activation quant: #40879
- [AMD][DSV4] moe: enable shared-expert fusion on the grouped-topk path (megamoe): #40943
- [AMD] Add tuned dsv4 shape: #40996
- [AMD] Add .co for deepseek v4 fp8 decode kernel and add group decode opt: #41120
- [AMD] Fix int32 offset overflow in Triton DSv4 KV store kernels: #41159
- [AMD][GLM5] Fuse shared expert into AITER MoE on gfx950: #41161
- [AMD] Honor an explicit triton moe_runner_backend for mxfp8 on ROCm: #41377
- dsv4.1-amd: gfx950 MXFP8 matmul kernels and fp8-grid producers: #41018
- dsv4.1-amd: KV cache layouts, FP4 indexer, compressor and router kernels: #41019
- dsv4.1-amd: gfx950 sparse decode attention and sorted top-k: #41020
- dsv4.1-amd: fused mHC boundary and all-reduce + mHC post kernels: #41021
NPU / Ascend
- feat(hicache): support Ascend Mamba states with FIA and async IO: #32500
- [NPU] decoding procedure optimization on qwen3.5/3.6: #35958
- [NPU] Fix xgrammar apply_vocab_mask device dispatch to use torch.ops.npu: #36120
- [NPU] add chunk gdn kernel and unify ssm state layout for ascend gdn backend: #36187
- fix(npu): fix hybrid KV transfer with PP prefill in PD disaggregation: #38402
- [NPU] Refactor weight processing and add NPUSwigluLimit activation: #38420
- [NPU][Fix] update low latency quantization input and update MXFP8 tests: #38831
- feat(npu): Support returning indexer top-k results: #39060
- [NPU][Diffusion] Optimize SenseNova-U1 batched generation: #39382
- [NPU] Avoid device synchronization in Ascend sampling: #39404
- [NPU] Support nccl backend for --remote-instance-weight-loader: #39413
- [NPU] Adapt hicache for K3 hybrid models: #39415
- dsv4(npu): support prefill context parallelism with interleave and zigzag: #39427
- [NPU][Diffusion] FA MXFP8 and modelslim w4a4f8 and w8a8f8 support for Wan2.2 and FLUX: #39438
- [NPU][Fix] Fix undefined require_attn_tp_allgather: #39561
- [NPU] support kimi k3 on A5 and improve performance: #39589
- [NPU] Support batch invariant FIA graphs for deterministic inference: #39607
- [NPU] fix npu docker workspace directory: #39732
- [NPU][DSV4] dsv4 enable cpp: #39820
- [NPU] Run arch35 block-FP8 dense linears on the native MXFP8 GEMM: #39823
- [NPU] Gate DFlash replay metadata refresh behind spec_algorithm check: #39879
- [NPU] Fuse MXFP4 W4A8 MoE gmm1 + swiglu + requant into one kernel: #39881
- [NPU] Avoid repeated BF16 wo_a weight transposes in DeepSeek-V4 decode: #39919
- [NPU][Fix] Avoid M-RoPE recompilation for variable sequence lengths: #40371
- [Fix][NPU] fix dp-attn hang when pin_mem is True on NPU: #40446
- [NPU] Fix DSV4 hard-coding kv dtype: #40510
- [NPU] Update CANN version to 9.1.0: #40524
- [NPU] support NPU 910C L2 memcache offload: #41527
- [Docs][NPU] Add MiMo-V2.5-Pro FP4 DFlash best practice on Ascend NPU: #40577
CPU / Intel / XPU
- [Intel XPU] Enable fused_moe_triton tuning on XPU and add tuned DeepSeek-OCR-2 configs: #28723
- Speculative Decoding with NGRAM support for XPU: #31362
- [CPU] Add fp8_per_tensor_scaled_mm_cpu kernel: #32618
- [XPU] Enable HiSparse hierarchical sparse KV cache on Intel XPU: #32792
- [CPU][Diffusion] Add fused scale-shift and norm kernels for CPU: #33452
- [Intel][XPU][KVCanary] Enable KV Canary on Intel XPU: #33520
- [Intel][XPU] Enable chunked prefill scnearios for XPU with UT: #33804
- [XPU] Use torch scaled_mm for XPU block FP8 linear: #35605
- [Diffusion] Fix the XPU capability gates that broke the Wan2.2 A14B DiT path: #36825
- [XPU] Qwen3.8-flash-next enablement: #37213
- [CPU] Upgrade torch cpu dependencies to version 2.14.: #37939
- [XPU] weekly simple model enablement 2026/09/14: #39439
- [CPU] Add fused_sigmod_mul_cpu operators to the Meta Muse Glimmer model.: #39538
- [Intel GPU] Xpu/weekly simple model enablement 2026 09 21: #40664
- [XPU] Support compressed-tensors W4A16 by reusing the torch int4pack path: #40828
- [XPU] Bump sglang-kernel-xpu wheel to v0.3.0: #41220
Security
- fix(openai): reject request-supplied chat_template by default: #28135
- Restrict SafeUnpickler standard-library globals: #39858
- fix: restrict SafeUnpickler to explicit globals: #40259
Cookbook Updates
- DeepSeek-V4: a PD-disaggregated arm for B200 FP4 agentic serving with DSpark (#40610); MI355X Pro PD pairs with DSpark and UMBP, plus MegaMoE, FP8 KV attention, and breakable CUDA graph recipes: #41024, #41458
- GLM-5.2: MI355X MXFP4 recipes move to HIP Top-K and newer images, and the high-throughput cell adds HiCache: #40148, #40570, #41109, #41597
- GLM-5.3 and GLM-5.3-Flash: reasoning and tool-call parsers on by default, and the DCP option is temporarily removed from GLM-5.3-Flash: #40497, #40036
- Qwen3.5: GB200 and GB300 configs, and the MI355X HiCache recipe uses the kernel IO backend with the
page_firstlayout: #40770, #39572 - Qwen3.8-Flash-Next: NVIDIA NVFP4 on B200, B300, and GB300: #41046
- Kimi-K3: an Ascend 950PR/DT recipe: #40575
- Intel XPU: the cookbook now marks XPU-supported models: #33649
Dependencies
- xgrammar 0.2.1 to 0.2.7: #39026
- cache-dit 1.3.0 to 1.5.1: #40104
- sentencepiece pinned to 0.2.1 (0.2.2 rejects null pieces in InternVL tokenizers): #41777
- sgl-eval 0.1.0 to 0.1.2: #40620
- CUDA 13.4 image:
nvidia/cuda:13.4.1-cudnn-devel-ubuntu24.04base, sgl-kernel 0.4.7, sgl-deep-gemm 0.2.0, sgl-deep-ep 0.1.2: #40987; Triton 3.8 compatibility for DeepSeek-V4.1-Flash on Rubin: #40805 - CPU: torch 2.14.0, torchvision 0.29.0, triton 3.8.0: #37939
- XPU: sglang-kernel-xpu 0.2.0 to 0.3.0: #41220
- ROCm: AITER pin moves to
acf8fdf9: #39965 - NPU: CANN 9.1.0 with Python 3.12 (#40524) and sgl-kernel-npu 2026.9.0.post5: #40157
- Official releases now also publish a versioned renderer image: #40639
Breaking Changes & Upgrade Notes
- The unified radix tree uses the Rust core by default. Configurations the Rust core does not support fall back to Python automatically; set
SGLANG_UNIFIED_RADIX_TREE_CORE_BACKEND=pythonto force the Python core everywhere: #39627 - FlashInfer MoE fused finalize is off by default for numerical accuracy. This costs some W4A4 MoE latency; set
SGLANG_FLASHINFER_MOE_FUSED_FINALIZE=1to restore the fused atomic finalize: #40105 - PD decode now defers KV release on abort by default on the Mooncake and NIXL backends, holding pages until every prefill rank acks (30 s timeout,
SGLANG_DISAGGREGATION_DEFERRED_DECODE_KV_RELEASE_TIMEOUT). SetSGLANG_DISAGGREGATION_DEFERRED_DECODE_KV_RELEASE=Falseto restore immediate release: #41023 - Request-supplied chat templates are rejected. A
chat_templatekey insidechat_template_kwargson/v1/chat/completionsor/v1/tokenizenow returns an error unless the server starts with--trust-request-chat-template; otherchat_template_kwargsare unchanged: #28135 - Deprecated endpoints, env vars, and aliases are removed. Use
/model_infoinstead of/get_weight_versionand/weight_version,/v1/loadsinstead of/get_load, andPOST /hicache/storage-backend/clearinstead of/clear_hicache_storage_backend.SGLANG_NSA_*aliases give way toSGLANG_DSA_*,SGLANG_ENABLE_SPEC_V2is gone,--elastic-ep-rejoinbecomes--elastic-ep-join-mode recover,--kv-cache-dtype fp4_e2m1becomesfp4_mx_block16, andpython -m sglang.bench_offline_throughput/sglang.bench_one_batchmove tosglang.benchmark.offline_throughput/sglang.benchmark.one_batch: #40795 - Legacy radix cache implementations are removed. The experimental C++ radix tree (
SGLANG_EXPERIMENTAL_CPP_RADIX_TREE) is gone (#40775), andSGLANG_ENABLE_UNIFIED_RADIX_TREEno longer does anything, though it still prints a deprecation warning (#40780). Code importingSWARadixCache,MambaRadixCache, orHiRadixCachemust move toUnifiedRadixCache: #40313, #40787 - Custom logit processors must return full-shape logits. A
CustomLogitProcessorthat returned a reduced shape such as(1, vocab)and relied on broadcasting now raises at decode time. The built-in processors are unaffected: #39234 - RL weight updates run inside a session.
update_weights_from_tensorandupdate_weights_from_distributedmust be called betweenbegin_weight_update()andend_weight_update().UpdateWeightsFromTensorReqInput.disable_draft_modelis replaced byselector="target", andChecksumInfo.parallelism_infois now a list with one entry per role: #40777 - Parallel-state getters are deprecated. They are no longer re-exported from
sglang.srt.distributed; read groups throughget_parallel()(for exampleget_parallel().tp_group) or import fromparallel_statedirectly: #40342, #40344, #40707 - Ascend A2 (910b) images are no longer published. Release and nightly images ship for A3 and 950 (A5) only, on CANN 9.1.0 with Python 3.12: #40524
New Contributors
- @fechaMe made their first contribution in #35809
- @renyiming25 made their first contribution in #35802
- @JustinPerlman made their first contribution in #37839
- @ironashram made their first contribution in #31926
- @yl3469 made their first contribution in #34012
- @yuzhouo7 made their first contribution in #35123
- @yekerr made their first contribution in #39680
- @Betelgeu made their first contribution in #38780
- @maithilijoshi20 made their first contribution in #35260
- @gurpreet-dhami made their first contribution in #35573
- @hmalgewatta made their first contribution in #37547
- @vvinniev34 made their first contribution in #38750
- @mindlab-bot made their first contribution in #38604
- @jmydurant made their first contribution in #38792
- @dmtri35 made their first contribution in #40184
- @salexspb made their first contribution in #39837
- @chaijiacheng888 made their first contribution in #38996
- @yuhanchan made their first contribution in #38740
- @chenyang08056032 made their first contribution in https://github.com/sgl-project/sglang/pull/38339
- @YZY00Raiser made their first contribution in https://github.com/sgl-project/sglang/pull/40392
- @harmya made their first contribution in https://github.com/sgl-project/sglang/pull/40217
- @ollybbmonster made their first contribution in #40448
- @divyvasal made their first contribution in #38932
- @xenshinu made their first contribution in #40201
- @Amrutha-M05 made their first contribution in https://github.com/sgl-project/sglang/pull/32792
- @skyler-apdx made their first contribution in #38786
- @kpjeeja made their first contribution in #28652
- @GungnirAP made their first contribution in #29189
- @RuibinCheung made their first contribution in #39775
- @avalliappan-nvidia made their first contribution in #32673
- @zhou-yuhan made their first contribution in #27265
- @zhujianwei-ops made their first contribution in #36120
- @jshn9515 made their first contribution in https://github.com/sgl-project/sglang/pull/40647
- @surajm20061998 made their first contribution in #39312
- @tomjen12 made their first contribution in #39525
- @siliangchen-amd made their first contribution in #40557
- @emre570 made their first contribution in #40175
- @wookjeHan made their first contribution in #40309
- @peilii made their first contribution in #40260
- @jshanson7 made their first contribution in #40468
- @MatsueYu made their first contribution in #35958
- @Apexsf made their first contribution in https://github.com/sgl-project/sglang/pull/39929
- @CjhHa1 made their first contribution in #34417
- @chunxiaozheng made their first contribution in #38652
- @weimin023 made their first contribution in https://github.com/sgl-project/sglang/pull/41150
- @TarzanZhao made their first contribution in #41223
- @arieller made their first contribution in #41292
- @nvcnvn made their first contribution in #39889
- @cuzmi made their first contribution in https://github.com/sgl-project/sglang/pull/41266
- @mrusanovsky made their first contribution in #37462
- @yonigozlan made their first contribution in #40477
- @floatlibai made their first contribution in #39060
- @Denji-kk made their first contribution in #39539
- @Hrithvik-Alex made their first contribution in #41588
- @zldrobit made their first contribution in #41059
Full Changelog: v0.5.20...v0.5.21
Highlights
713 PRs from 237 contributors.
New models in this release (see the cookbook for all supported models):
| Model | Type | PRs | Cookbook |
|---|---|---|---|
| GLM-5.3-Flash | Autoregressive | #36507, #38621 | link |
| Hy4-Preview | Autoregressive | #36805 | link |
| Qwen3.8-Flash-Next | Autoregressive | #37500 | link |
| K2 Horizon | Autoregressive | #37654, #38033 | link |
| Nanbeige4.2 | Autoregressive | #32151 | |
| SenseNova-U1.5-8B-MoT | Diffusion | #36606 | link |
| FastH3 (4-step MiniMax-H3 distill) | Diffusion | #37480 | link |
| VDN-H3 (hybrid-attention MiniMax-H3 distill) | Diffusion | #37903 | link |
Sampling masks for RL rollouts. With return_sampling_mask, each decode step returns the exact token support the sampler drew from and the log-probability of the sampled token under it, so a trainer can replay the rollout without reconstructing top-k or top-p (#36630). Masks now run under overlap scheduling: on Qwen3-8B, decode throughput is 17% higher at batch 1 and 52% higher at batch 64 than the previous implementation. Capacity is set by --sampling-mask-max-tokens (default 4096) (#36631). DisallowedTokensLogitsProcessor is supported alongside masks (#38279).
Unified radix tree. Branching-point caching for the SWA component keeps the sliding-window state at the point where requests fork from a shared prefix, so branches reuse it instead of recomputing. On DeepSeek-V4-Flash with a shared system prompt, token hit rate rises from 43.8% to 60.8% and mean TTFT falls from 1.57 s to 1.07 s (#34565). An opt-in external linker lets the tree address a shared global memory pool through Mooncake or UMBP (#37381).
DSpark under PD with decode context parallelism. A DCP1 prefill can now transfer its DSpark draft KV to a DCP-N decode, so hybrid models such as Kimi-Linear run DSpark in disaggregated, context-parallel serving. Verified on 8x B300 over NIXL and Mooncake up to 256K input (#37709).
Responses API storage is opt-in. /v1/responses no longer retains results in memory unless the server starts with --enable-response-store. Without it, retrieval, previous_response_id chaining, and background requests return 400; PD deployments cannot enable it (#39122).
SGLang Simulator. A CPU-only simulator runs the real scheduler, radix cache, and hierarchical cache with a latency predictor in place of the model forward. Against measured serving traces it predicts TTFT within about 6% on most traces (up to 10% on the longest 32K to 128K ones) and prefix reuse within 0.05 percentage points, for cache and scheduling studies without GPUs (#33824).
Prefill context parallelism v1 removed. The strategy-based implementation is now the only prefill CP path; the v1 runtime and its CLI options are gone. Prefill CP on HIP, NPU, and MUSA is rejected until those platforms are ported (#36228).
Faster model loading on ROCm. Large pageable host-to-device copies are staged rather than pinned in place, which stops the driver from suspending GPU queues on every eviction. GLM-5.2 at TP4 on 4x MI355X loads in 40.4 s instead of 505.7 s (#37720).
Intel XPU joins the release images. Every tagged release now builds and publishes lmsysorg/sglang:vX.Y.Z-xpu from the XPU Dockerfile, so Intel GPU users get a versioned image instead of relying on nightly builds (#37340).
DeepSeek-V4 on Blackwell. TRT-LLM attention kernels now cover DeepSeek-V4's CSA and HCA layers on SM100 and SM103: about 1.2x faster prefill and 1.45x faster decode than FlashMLA at the kernel level on B200 (#30805). FlashInfer MegaMoE is available as --moe-runner-backend flashinfer_megamoe; on DeepSeek-V4-Flash NVFP4 at TP4/DP4 it adds up to 11.9% prefill throughput at 8192 tokens per rank, with decode within 2% of the trtllm runner at saturation (#31470).
DeepSeek-V4 on RTX PRO 6000. On SM120 the sparse-MLA indexer now runs on DeepGEMM's paged-MQA kernel and the DeepGEMM FP4 MoE backend is enabled, replacing the torch fallback that was the only working path. On 4x RTX PRO 6000, DeepSeek-V4-Flash decode TPOT drops from 36.1 to 10.5 ms at batch 1 and TTFT falls 20% from 8K to 128K input. Opt-in through the environment flags in the PR (#29927).
Dependencies and images. The CUDA 12 lane is retired; v0.5.19 was the last release with -cu12x wheels and images (#38404). sglang-kernel moves to 0.4.7 (#39346) and sgl-deep-gemm to 0.2.0 (#39371). New images: ROCm 10 for MI30x and MI35x with a matching kernel wheel (#38763), gfx1151 for Strix Halo / Ryzen AI MAX+ (#33939), and Moore Threads MUSA (#36709). ROCm 7.0 CI, images, and kernel wheel are retired (#38632, #38767).
Full release notes by category below; breaking changes are at the end.
Speculative Decoding
- [Speculative Decoding] Add native UNO serving support: #37667
- feat: add optimized Domino rollout to DFlash V2: #36899
- feat: support TP>1 Domino rollout for DFlash V2: #37069
- [Spec] Stage Inkling MTP draft metadata before verify: #38169
- [Spec] Allow speculative workers to stage prefill shared reads: #38554
- [Spec] Support large MTP batches in short-convolution metadata: #38558
- [EAGLE] Prune draft-extend logits to selected rows (+2.5% output tokens/s/GPU at 102K context per rank): #35546
- [KDA] Support ReplaySSM ring-write in the fused chain-verify kernel: #36821
- [GDN] Amortize ReplaySSM checkpoint materialization: #35544
- perf(gdn): select ReplaySSM verify loop unrolling by shape: #36970
- Allow custom policy for adaptive speculative decoding: #37274
- Support speculative decoding with unified SWA memory: #36403
- Improve CUDA graph and speculative execution output handling: #37329
- Fix DSpark CUDA graph replay with MegaMoE TP attention: #34919
- [Fix][Mamba] Clear deferred init metadata before speculative decode: #37165
- [Fix] Track DFlash Mamba state at checkpoint boundaries: #37818
- [Fix] Load Qwen3.5 MTP embedding under PP: #37471
- fix: stop shadowing the DSpark shared-experts fusion guard: #39366
- [Profiler] Label draft-runner steps DRAFT and target verify VERIFY in step spans: #38630
Piecewise & Breakable CUDA Graph
- [Memory] Size the CUDA graph pool from warmup measurements and fix graph-pool borrowing: #36911
- [Memory] Retire graph borrow pool before updating static runs: #37966
- [Memory] Reuse output storage across full prefill CUDA graphs: #38038
- [Unified Memory] Enable prefill cuda-graph capture: #37418
- Decouple ragged CUDA graph request and token capacities: #37300
- Reuse live CUDA graph executables during dedup registration: #39176
- Scope graph-pool borrowing to the runtime and reduce fragmentation: #39177
- Expose a capacity check for graph-pool borrows: #39178
- Keep graph-pool borrows on their allocation stream: #39180
- Add a provider hook for prefill-buffer ceilings: #39182
- [Refactor] Generalize attention graph variants in the decode runner: #38993
- [Fix] Coordinate FullCG prefix variants across DP ranks: #37888
- [Fix] Keep a shared MAX_LEN prefill CUDA graph bucket when the graph captures a DP gather (MegaMoE sparse-DP hang): #37933
- [Fix] Key CUDA graph dedup signatures on kernel function identity: #37657
- [Bug] Include DSA variant in exact-bucket graph admission: #38992
- Fix KV-canary workspace accounting after graph capture: #38596
Attention Backends
- [DSv4] Integrate TRT-LLM DSv4 Attention for SM100/103 (kernel-level prefill about 1.2x and decode about 1.45x faster than FlashMLA on B200): #30805
- [DSV4] Support raw-index output in TopK v2: #33672
- [DSV4] hc-prenorm: fuse the combine step into a Triton kernel: #35118
- [FlashInfer] Avoid D2H sync for sliding-window lengths (TPOT 5.4 to 5.1 ms on 8x A800): #32218
- [Attention] Size FlashInfer MLA indptr buffers to the padded max batch: #38590
- [Perf] trtllm_mla: reuse the fused fp8 KV/Q prepare on target verify: #39232
- [Performance] Optimize Qwen3.5 GDN prefill projection layouts: #36267
- Fix Qwen3.5 GDN multi-item scoring (+18.8% throughput, P50 latency -16%): #33922
- Support NoPE layers in the tokenspeed_mla FP8 prefill hook: #38152
- fix(qsa): make the paged sparse-decode gather memory-safe (zero-fill scratch, int64 offsets, dequant FP8 on gather): #38851
- fix(qsa): dequantize FP8 cached prefixes in the sparse prefill kernels: #38855
- Reland fix(qsa): clamp the compress gather to the rows (#38346): #39446
- [Qwen 3.8 Next] reuse old cuda stream instead of endlessly creating streams: #39474
- [Fix][DSA] Bound prefill Triton specializations for page-table stride: #37093
- [Fix] Wait for PDL before reading DeepSeek V4 K cache locations: #38409
- [Fix] Fix Llama 4 FA3 local attention with paged KV cache: #32902
MoE & Expert Parallelism
- [NVIDIA] Support flashinfer Mega Moe (prefill input throughput up to +11.9% vs the trtllm runner on DeepSeek-V4-Flash NVFP4 at 8192 tokens per rank): #31470
- [MoE Refactor] Migrate SM100 trtllm-gen mxfp4 MoE onto MoeRunner: #32405
- [Deepseek V4] Keep fp32 routing weights in the mxfp4 trtllm MoE: #33608
- Keep fp32 routing weights in the fp8 block-scale and bf16 trtllm MoE: #33631
- Drop the routing bias casts in flashinfer trtllm MoE: #33591
- Fix DeepSeek-V4 routing: sqrtsoftplus underflow and unfloored renorm: #34459
- keeping router GEMM in fp32 for deterministic inference (DeepSeek V3/V4): #38176
- Fuse Nemotron latent MoE projection and shared add: #30430
- [Kernel] Replace dsv3_router_gemm with the unified tiny GEMM: #34693
- [Kernel] Avoid zero-bias allocation in fused softmax routing: #36811
- [Kimi-K3] Accept fp32 routing weights in the fused MoE finalize: #38612
- Cast fp32 routing weights to bf16 in the Kimi-K3 fused finalize: #38588
- Use fp32 in TRTLLM all reduce buffers: #36143
- [Comm] Drop the in-tree MNNVL CuTe DSL port in favor of FlashInfer 0.6.18: #37206
- [Moe] Fix flashinfer_trtllm silently dropping swiglu_limit clamped SwiGLU activation (GLM-5.3-Flash FP8 GSM8K 0.928 to 0.936, matching the Triton runner): #39920
- Fix MegaMoE buffer allocation and caching for effective SM budgets: #39223
- Fix native MoE handling of noncontiguous top-k IDs: #36407
- [Fix] Accept int64 top-k IDs in FlashInfer routed MoE packer: #32882
- fix(moe): cast filtered-activation expert_ids to int32 for torch.compile: #38085
- fix(gpt-oss): avoid duplicate MoE reduction with DP attention: #37199
- Fix mooncake scale joiner groups: #38503
Quantization
- [Quant][ue8m0 fix] group requant_weight_ue8m0 reduce reserved gpu memory: #31755
- [1/N] Quantization Refactor: remove dead code and dedup the FP4 marlin helpers: #37552
- Delete cutlass_mla, non-Marlin GPTQ, AWQ AOT kernel, and Dual Chunk Flash Attention: #32114
- fix(modelopt_fp4): skip NVFP4 swiglu-fusion interleave for shared experts with swiglu_limit: #37378
- Disable Hopper GLM shared-expert fusion for modelopt_fp4 Marlin: #37325
- [Fix] Preserve FP32 in SM107 MXFP8 fallback: #37489
- Fix block-scale swizzling device placement: #37849
- Keep NVFP4 blockscale swizzle padding on the input device: #39141
- Fix dummy initialization of inverse weight scales: #35491
- [Chore] harden checkpoint quantization metadata parsing: #36922
Parallelism & Disaggregation
- [PD] Transfer the DCP-replicated DSPARK draft KV in DCP1->DCP-N relayouts: #37709
- [Qwen3.8-Next] Add PD state transfer for Flash Next: #36651
- [Intra-node PD][DSV4] Pack all layers into one batch for INTRA_NODE_NVLINK path (KV transfer latency 178 to 119 ms, E2E latency -20% on a 2P1D B200 node): #38984
- [PD] Share the prefill->decode failure notification across backends: #36612
- [PD] Gate deferred decode KV release on backend capability: #37454
- [PD] Bound transfer engine init with
SGLANG_DISAGGREGATION_ENGINE_INIT_TIMEOUT: #37874 - [PD] Preserve the prefill rank during rebootstrap: #39357
- [PD] Optimize paged allocator free-list release: #37146
- [PD][LoRA] Gate decode admission on adapter slots: #39332
- [Unified Memory] PD disaggregation for every unified pool shape: #37506
- fix(disagg): poll receivers during decode preallocation: #37483
- Fix DSA compression tail capacity for PD decode request slots: #38417
- fix(glm-5.2-nvfp4): bound Mooncake synchronous transfer batches: #32758
- [DCP] Allow fi_a2a on single-node systems Blackwell without MNNVL fabric ( ex B200 B300) (median TPOT 1.5 to 5.5% lower than the NCCL all-to-all on 8x B200): #37767
- [DCP] Resolve --dcp-comm-backend to fi_a2a/a2a by default for every model: #39165
- [Feature] Unified memory: support decode context parallelism for Kimi-Linear: #36890
- [Feature] Unified memory: support decode context parallelism for the trtllm_mla family: #37693
- [Perf] Unified memory: close the DCP decode gap on Blackwell: #37926
- [Fix] Merge adjacent KV-row frees so a mid-page split under DCP cannot double-free: #38941
- [Fix][DCP] Localize widened KV ids in MLA retraction CPU backup/restore: #39487
- [CP V1 Deprecation 2/5] Make strategy prefill CP canonical: #36223
- [CP V1 Deprecation 3/5] Remove generic prefill CP v1 runtime: #36228
- [CP V1 Deprecation 3.5/5] Deprecate HIP/NPU/MUSA prefill CP and remove legacy implementation: #38293
- [CP V1 Deprecation 4/5] Canonicalize prefill CP API names: #36229
- [CP V1 Deprecation 5/5] Update prefill CP documentation: #36230
- fix: gather CP-sharded tokens before TP-sharded dense MLP under prefill CP: #38078
- Fix inflated row pitch when a CP round-robin shard has a single row: #34142
- [SP] Make attention-TP sequence sharding a per-forward batch property: #37546
- [Fix] Stamp sequence-parallel state on dummy forward batches: #38564
- [DP Attn] Fix crash for no token all-gather case: #39899
- [Fix] Disable NCCL graph buffer registration for the TP LM-head all-to-all (pure-DP decode hang under request bursts): #38936
- [Misc] Extract PP dynamic chunk sizing into a
DynamicChunkSizerscheduler component: #37674 - [Fix] Lock PP dynamic-chunk profiling requests before releasing through the tree cache: #37509
- [Fix] Apply the attention-CP broadcast result in PP dynamic-chunk profiling: #37669
- [Fix] Broadcast PP dynamic-chunk profiling failures so every rank disables together: #37675
Scheduler & Runtime
- [Scheduler] Add HRRN schedule policy to significantly reduce TTFT (
--schedule-policy hrrn; mean TTFT -69% and p99 -8% vs FCFS on a GLM-5.2 production trace): #32911 - [Scheduler] Make request-timeout aborts rank-consistent to fix TP collective hangs: #37143
- [Scheduler] Count the parked chunked-prefill request in the busy mem check: #37502
- [Scheduler] Unify per-iteration request intake into ingest_requests(): #38389
- Fail fast on undersized swa pool: #37610
- Reduce tokenizer overhead and offload CUDA VMM publication: #37330
- Keep VMM capability votes on CPU: #38722
- state_capturer: pin the exact host-cache size via mmap + cudaHostRegister: #37285
- Generalize auxiliary outputs: #39164
- [Model Loader] Split weight loading from postprocessing: #34981
- Fix gpt-oss RunAI streamer weight ownership: #38908
- Fix RunAI object-storage checkpoint index filtering: #38988
- Clean logging under --weight-loader-prefetch-checkpoints: #33930
- [Fix] Release NCCL on scheduler exit and let the ASGI server own shutdown: #39560
- [Fix] Aggregate all IPC weight update responses: #39534
- [Fix] Avoid scanning crash-dump token buffers during GC: #37297
- add reindex_device_id to device OOT plugin: #36220
- [Metrics] Add rolling scheduler utilization counters: #37461
- [Metrics] Export scheduler stage wall time: #37636
- [Metrics] Report logical prefill token counts: #38566
- [Observability] Fix missing e2e/decode/inference latency span attributes: #37789
- [Ray] Support Ray metric backend for engine metrics: #31415
- [Profiler] Add SGLANG_PROFILE_BY_STAGE_DECODE_MIN_BS to defer the decode-stage capture: #38067
- Add num_prealloc_ready_tokens to decode load snapshot: #38065
- [Bench] Support real-traffic replay with early-stop-aware steady-state metrics in bench_one_batch_server: #37469
- [Benchmark] Add an opt-out for the token-capacity check: #39284
- Fix reasoning metrics and add TPOT to bench_multiturn: #35443
Sampling
- [Sampling] Capture masks from sampler support: #36630
- [Sampling] Support sampling masks with overlap scheduling (Qwen3-8B decode throughput +17% at batch 1 and +52% at batch 64 vs the previous non-overlap path): #36631
- [Sampling] Allow sampling-mask replay with DisallowedTokensLogitsProcessor: #38279
- perf: use Gumbel-max trick in the main sampler to cut decode CPU dispatch (Qwen3.5-2B decode +75% at batch 1 and +23% at batch 32): #38117
- Fix custom logit processor params when num_tokens_in_batch is used: #38730
HiCache & Radix Cache
- [Unified Tree] Support Branching-Point Caching for the SWA Component (shared-prefix token hit rate 43.8% to 60.8%, mean TTFT 1.57 to 1.07 s on DeepSeek-V4-Flash): #34565
- [Unified Tree] Port SWA Branching-Point Caching to the Rust TreeCore: #37584
- [Unified Tree] fix: make dfs weight ordering iterative: #38313
- [Unified Tree] Preserve aux LRU recency when splitting nodes: #38482
- [Unified Cache][5/N]: Integrate external linker mode end to end: #37381
- [Unified Cache][6/N]: Add UMBP external linker: #37578
- [Unified Cache][7/N] Support MTP, EAGLE, and DSpark draft KV caches in the external linker: #37914
- [Unified Cache][AMD] Support DeepSeek-V4 unified KV in direct external linkers: #38269
- [Rust TreeCore] Harden runtime and CI parity: #37303
- [Rust TreeCore] Support external cache linker: #37306
- [Rust] Rename mem-cache to sglang-radix-tree: #37290
- [KV Shard 1/4] Logical-page placement with UnifiedRadixCache: #38356
- [Cache] Forward fast prefix matching capability: #37844
- [Refactor] Let eviction policies take construction parameters: #37795
- [Perf] Walk the radix tree by offset instead of re-slicing token storage (ported from #36507): #37324
- [Perf] Vectorize alloc_extend_naive to remove the per-request Python loop: #37938
- [Radix Cache] Fix PureSWA tail release without insertion: #38349
- fix: preserve SWA host lock on node split: #38138
- fix: collect prefix hash values iteratively: #38204
- fix: align write-through pending across tree cores: #37278
- [HiCache] Add MLA host-dedup primitives: #36800
- [HiCache] Replace skip_lock_node_ids with a segment lock protocol: #36848
- [HiCache] Buffer mode support sidecar pool: #37424
- [HiCache] buffer mode: anchor-lock staged prefetches by default: #37464
- [HiCache] L3 storage prefetch lifecycle metrics and cross-tier attribution fixes: #37503
- [HiCache] Count hit allocations and in-flight backups in the buffer pipeline idle check: #37883
- [HiCache] Fix side pools to use resolved host allocator: #38350
- [HiCache] Account for newly pinned ancestors in load-back quota: #38481
- [HiCache] Release buffer prefetch anchor locks during storage cleanup: #38483
- [HiCache] Publish a host store event for storage-prefetch refills: #38486
- [HiCache][LoRA] Isolate storage pages by extra key: #38577
- [HiCache][LoRA] Simplify decode offload hash inputs: #39162
- [HiCache] fix: preserve SWA host lock boundaries across splits: #38835
- [HiCache] Keep the file backend temp file name within NAME_MAX: #38925
- [HiCache] Rework the buffer-mode storage prefetch pipeline and retry bookkeeping: #39283
- [HiCache] Optimize buffer-mode storage existence bookkeeping: #39480
- HiCache: Add @rank_consensus to various functions: #37425
- HiCache: Reduce the number of
all_reduceincheck_hicache_eventsfor PP (TTFT -7% on DeepSeek-V4-Flash with HiCache L3, PP4 TP2): #37562 - Fix buffer-mode idle tracking and VLM memory sizing: #37567
- fix: keep queued Mooncake linker loads after abort: #38195
- fix(nixl): make FILE path-mode devId globally unique: #34362
- Scope prefetch cache state to the request attempt: #39318
- [Fix] HiCache startup ImportError on the pinned kernel wheel: #39516
- [Mem Cache] Make
free_swasync-free onpage_size == 1: #36723 - [Mem Cache] Drop the
torch.uniquesync from the SWA page expansion: #37463 - [Mem Cache] Split duplicate insert frees at the SWA eviction floor: #37481
- [Mem Cache] Require page-aligned starts in
free_segmentand drop the boundary trim: #37729 - [Mem Cache] Route hybrid SWA full-side kv-row frees through
free_segment: #37876 - [Mem Cache] Move the unified-memory allocators into
allocator/and split the composites out: #38072 - [Mem Cache] Free hybrid SWA pages by one representative per page on
page_size > 1: #38159 - [Mem Cache] skip duplicates host evict via environ: #38462
- [Misc] Resolve SWA ownership at enqueue time for grouped free(): #36646
- Fix unified SWA: size a non-owner's v2p by the id space it must address: #37560
- Size the unified read-table grid from bs, and fuse the allocator's tombstone scatters: #37511
- Build the unified read stream directly, without the page-table rectangle (decode CUDA graph memory 2.89 to 1.89 GB on Kimi-Linear-48B TP2): #37512
- fix(unified-memory): evict Full KV for Mamba byte shortfalls: #36713
- Store mamba prefix-cache checkpoints at the configured SSM state dtype: #34820
- Fix mamba radix cache ssm state indexing: #37836
- [Fix] Vacuous marker writes in the cache tests, and an undebited Mamba admission slot: #36415
LoRA
- [LoRA] Support MoE in full and breakable prefill CUDA graphs (prefill throughput +39.5 to 40.9% with Triton LoRA on Qwen breakable graphs, GB300): #38578
- [Intel][XPU][LoRA] Enable LoRA on Intel XPU: #30345
Multimodal
- [Perf] Optimize Qwen3-VL unique-image serving on H100: #36411
- fix(multimodal): handle tensor images in exact-token preprocessing: #30368
- Fix multimodal embedding cache retaining full batches through views: #39120
- [Fix][Qwen-VL] Normalize
<image>sentinel on artifact fast path: #39278 - fix(glm4v): disambiguate mixed image video offsets: #37971
- [Fix] Alpha-channel images and tool-result media ordering (port of #36507): #37320
- [MM] Add flag to force Kimi image preprocessing onto CPU: #39148
- [VLM] Retire aborted disaggregated prefill results: #36988
- fix(vlm): contain EPD request lifecycle failures: #36944
- fix(vlm): harden EPD receiver validation and liveness: #36945
- fix(vlm): make EPD cache publication transactional: #36949
Model Support & Optimizations
- [Cosmos3] Add cosmos3 Reasoner to llm only inference: #33572
- Support NemotronH_Omni_Reasoning_V3 in SGLang: #35599
- Inference Support Mamba 2 and 1 (Mamba-Codestral-7B, Falcon-Mamba-7B, and the state-spaces Mamba checkpoints): #34556
- [GLM-5.3 Flash] Restore and enable KPool metadata fusion: #38845
- MiniMax-M3: share the sparse index top-k across layers and reuse the decode top-k buffer (output throughput +6.4 to +20.6% at 80K input on 8x MI350X): #36527
- MiniMax-M3: Triton split-K router GEMV with in-kernel fixup: #36557
- perf(lfm2): fuse gating and short convolution on SM90: #37622
- [GraniteMoE] Load split per-expert quantized MoE weights: #37679
- Fix Muse Glimmer ModelOpt mixed weight mapping: #37510
- [Fix] Fix Spark2.5 hybrid SWA config: #37335
- fix(mamba): unify causal_conv1d col* dtype to x (MiniCPM-V-4.6 GDN prefill bf16/fp16 mismatch): #38039
- [Refactor] Clarify DeepSeek V4 metadata names for V4.1: #38947
- [Refactor] Generalize DeepSeek V4 compressed pool management: #38954
Kernel Library
- [Kernel] GLM 5.3 Flash related kernels (ported from #36507): #37477
- [Kernel] Register merged diffusion agent kernels with KDA backend: #37385
- [Kernel] Add fused MoE Triton configs for Qwen3.8-Flash-Next FP8 on NVIDIA H200 NVL (TP2+EP2) (1.14 to 1.23x at decode-sized batches): #38116
- [Kernel] Add tuned LFM2.5 Triton MoE configs on B300: #37158
- [Kernel] Drop the vendored dense BF16 GEMM port in favor of FlashInfer 0.6.18: #38124
- [Kernel] Add OOT dispatch for clamp position: #38687
- [JIT] Port the expert-pack MXFP4 kernels to load_jit and fix their launch limits: #38830
- feat(kernels): generalize persistent CuTe JIT cache: #33911
- Add SGLANG_CRASH_ON_JIT_COMPILE to forbid on-the-fly JIT compilation: #36615
Configuration System
- [Config] Round 6.1: "unset" gets its own spelling, and the declaration says what it means: #38046
- [Config] Round 6.2: the field declarations move to their namespaces, and the record is assembled from them: #38047
- [Config] Round 6.3: the record remembers how it was asked for, and is sealed while resolution runs: #38048
- [Config] Round 6.4: the runtime reads the bags, not the record: #38049
- [Config] Round 6.5: a namespace declares what it derives, next to what it derives it from: #38113
- [Config] Retire get_global_server_args, and clear the deprecated flags that have a replacement: #38375
- [Config] One writer for the declaration stash; no exception to the write seal: #38752
- [Config] msgspec.Struct for the config tier: #38753
- config: an out-of-tree replacement point for every resolution-pipeline step: #39134
- Fix dataclasses.asdict on the msgspec ServerArgs: #38958
OpenAI-Compatible API
- [Feat][Responses API] Support custom tools, encrypted reasoning replay, developer tier and model validation: #38690
- [PD][OpenAI] Gate /v1/responses persistence behind --enable-response-store, default off: #39122
- [OpenAI] Propagate PD routing metadata through /v1/responses: #35503
- [Responses] Fix empty-prompt routing for token-first chat encoders (kimi_k3, inkling): #35486
- [Feature] Add response-level input/output token ids to chat completions via SglExt: #34488
- Support non-strict GLM47 tool calls with EBNF constraints: #38890
- [Fix] Preserve GLM tool argument types across JSON Schema unions: #39136
- Auto-detect GLM-5.3 chat templates as glm45/glm47 parsers: #38297
- Force reasoning mode for GLM-5.3 chat templates: #39227
- Add granite_thinking_parser reasoning parser for Granite 4.2: #38693
- [Kimi K3] Rework skipped-think fix as opt-in force_nonempty_content with streaming coverage: #34187
- [Kimi-K3] Recover the reply when the model skips the think channel: #37743
- [Session] Fix session idle timeout after rejected requests: #39035
- [Session] Work with PD and Fix empty continuations: #39038
- [Session] Fix image append positions and parent metadata: #39145
- [Session + MM] Fix text positions in session continuations: #39144
- feat(agent sessions): attribute stored KV cache blocks to sessions: #37482
- Fix /model_info serialization when a config value is a class: #39237
- Fix: abort handling for dispatched requests after client disconnect: #35255
Rust Server
- [Router] Add composable scoring and eligibility policies: #37731
- [Router] Add load-aware prefill admission and bounded policy proposals: #37843
- [Router] Add bucket-aware policy domains and native cache indexing: #38108
- [Router] Publish cache-aware load state: #38139
- [Router] Preserve global cache affinity with bucket routing (TTFT p50 -18.5% vs power-of-two bucket routing): #38814
- [Router] Honor KV-event storage tiers in the cache-aware tree (1/4): #39108
- [Router] Expose the KV storage-tier stream and tree occupancy on /metrics (2/4): #39109
- [Router] Stream outcome observability for 2xx SSE streams: #38737
- [PD] Add /v1/responses support to the HTTP PD router: #36141
- [Rust] Gate health on startup warmup completion: #37994
- [Rust] Use Dynamo native renderers when chat templates are missing: #38939
- [Rust Server] Use node-local HTTP ports for DP attention: #34430
- [Rust Server] fix p/d bootstrap across dp listeners: #36234
- Rust server: align launcher and request validation behavior: #37327
- Allow CUDA VMM feature transport with the Rust frontend: #39347
- [gRPC] Stream engine state changes: #39915
- [gRPC] Expose native pause status: #37488
- [Fix] Seed raw tokenizer_path for smg-grpc-servicer in gRPC mode: #39105
- [SGL Router] Prepare dynamo-render dependencies: #39457
Simulator
A CPU-only inference simulator under tools/sglang-simulator that reuses the real scheduler and cache stack with a latency predictor in place of the model forward.
- [Simulator] Add high-fidelity CPU-based inference simulator: #33824
- [Simulator] Give the OFFLINE/BLOCKING comparison tolerances real headroom: #38732
SGLang-Diffusion
- [Diffusion] Add MiniMax H3 cube sparse attention (denoise 19.3% faster with the aggressive schedule on 2x H100): #34893
- [Diffusion][MiniMax-H3] Add SM90 Sage compute for SubBlock sparse attention (generation time -6.3 to -20.9% vs BF16 SubBlock): #37982
- [Diffusion] Add SpargeAttention backend: #37437
- [Diffusion] Add request-scoped Skip Softmax attention: #37959
- [Diffusion] Pick the attention backend by measuring it: #38689
- [Diffusion] Admit explicit attention backends by capability: #37441
- [Diffusion] Add cumulative extra-high quality tier: #37422
- [Diffusion] Support MiniMax-H3 Singularity hybrid checkpoints: #38455
- [Diffusion] MiniMax-H3: tiered AdaLN plan cache (pinned-host tier + per-plan LRU): #37266
- [Diffusion] MiniMax-H3: warm up at the served clip shape: #37945
- [Diffusion] MiniMax-H3: Only use fused qk_norm on NV: #33994
- [Diffusion] Fix MiniMax H3 WebUI inference settings: #36320
- [Diffusion] fix: stabilize H3 reference audio across repeated requests: #38225
- [Diffusion] Return Qwen-Image-Layered outputs and preserve CFG2 rounding (engine E2E 34.2 s on one H200 to 17.3 s on two, byte-identical layers): #38549
- [Diffusion] Optimize Qwen-Image-Edit attention on Hopper: #38584
- [Diffusion] Port the Wan VAE decoder fast paths to the Qwen-Image VAE: #38020
- [Diffusion] Keep the Wan VAE decoder channels_last and add a Triton NHWC nearest upsample: #38182
- [Diffusion] Fuse FLUX.2 NVFP4 FC1, SwiGLU, and FC2 quantization: #37096
- [Diffusion] Enable lossless BCG for FLUX.1-dev: #38591
- [Diffusion] Enable breakable CUDA graph for JoyEcho: #38110
- [Diffusion] Optimize LTX-2 QKNorm and split RoPE on Hopper: #38396
- [Diffusion] Preserve BF16 rounding in Hopper LTX QKNorm and RoPE fusion: #38533
- [Diffusion] Support diffusion decoder parallel tiling for LTX-2.5: #36026
- [Diffusion] Fuse LingBot per-token gated residual and RMSNorm modulate: #37910
- [Diffusion] fuse LingBot MoE group-limited top-k index selection: #38044
- [Diffusion] add Helios per-token gated-residual fusion (quality-gated): #38042
- [Diffusion] Fuse LongCat Image normalization and modulation: #38530
- [Diffusion] Optimize SANA-WM convolution post-processing and streaming GDN: #38529
- Cosmos3 fp8 mixed precision: #36380
- [Diffusion] Support mixed INT8 embeddings and Comfy NVFP4 encoders: #38506
- [Diffusion] auto-keep video DiT resident on high-memory GPUs (SANA-WM end to end -38% on one B300): #38001
- [Diffusion] Add explicit snapshot-offload component residency: #38535
- [Diffusion] Stream mapped weights on a shared host/device pool (layerwise-offload request 478 to 302 s): #38441
- [Diffusion] Read mapped layers directly when the host cannot cache them: #39022
- [Diffusion] Measure warmup memory and layer usage per phase for residency calibration (1/4): #37916
- [Diffusion] feat: spill large tensors over shared memory like numpy arrays: #38656
- [Diffusion] Set an explicit x264 preset for video output: #38657
- [Diffusion] Compose third-party component bundles safely: #37816
- [Diffusion] Consolidate plain state-dict component loaders: #38128
- [Diffusion] loader: reuse plain state-dict loading without per-model classes: #38127
- [Diffusion] loader: filter duplicate precision variants across custom loaders: #37616
- [Diffusion] Remove component loader capability switches: #36824
- [Diffusion] Key the VAE decode-dtype store by module layout: #38496
- [Diffusion] Serve requests that turn CFG off on a CFG-parallel server: #39303
- [Diffusion] Make the SP sequence gather pass contiguous shards: #39291
- [Diffusion] Don't route an unreadable checkpoint into the native fallback: #39292
- [Diffusion] Recover IPC JIT initialization after interrupted builds: #39034
- [Diffusion][Refactor] Refactor and unify RoPE execution for DiT models using RotaryEmbedding based on CustomOp: #33555
- [Diffusion] feat: add maybe_record_function profiler spans for request phases: #35922
- [Diffusion] Improve BCG warmup frame-count diagnostics for video models: #37890
- [Diffusion] Fix #38166: Restore non-layer placeholders before releasing host copies: #38171
- [Diffusion] fix: preserve mapped courier tensor lifetime: #37965
- [Diffusion] fix host-resident vocab tables loaded on GPU: #38012
- [Multimodal Gen] feat: support key masks on USPAttention's replicated-prefix path: #36735
Local & Desktop AI
- [SM120] DeepSeek-V4: DeepGEMM paged-MQA indexer +FP4 MoE+ page-split (decode TPOT up to 3.4x faster and TTFT -20% on 4x RTX PRO 6000): #29927
- [SM120] Use exact query-head widths for DeepSeek-V4 sparse MLA decode: #36655
- [Kernel] Add KDA NVFP4 GEMM for Qwen3.x on SM120 (1.32x geometric-mean kernel speedup at decode shapes on RTX PRO 6000): #36865
- [Kernel] Add KDA FP8 skinny GEMM for SM120: #38082
- [FP8] SM120: route FP8 linear to per-tensor (cudnn/nvjet) instead of channelwise cutlass (Ministral-3-14B decode 46.9 to 86.6 tokens/s at batch 1): #38006
- sm120 32GB mem-tier: raise decode cuda-graph max_bs 24->48 + chunked_prefill 2k->4k: #37898
- [Qwen3.8] Enable NVIDIA NVFP4 on DGX Spark with file-backed PLE and PDL router fix: #39126
- [Diffusion][minimax-h3] Add SM120 support for SubBlock sparse attention (1.22 to 1.85x lower latency on 8x RTX PRO 5000): #37332
- [Diffusion] MiniMax-H3 VAE decoder: unfused w2 bias on SM12.x (cuBLAS 16x16 kernel mis-dispatch): #37835
- [Diffusion] Pin layerwise host stores in place at their exact size: #39021
- [Fix] Register triton.runtime.cache.triton_key in the MPS stub so torch.compile keeps working: #37937
AMD / ROCm
- [ROCm] Stage large pageable H2D copies instead of pinning them in place (GLM-5.2 TP4 load on 4x MI355X from 505.7 s to 40.4 s): #37720
- [AMD] Optimize Kimi-K3 Triton MLA prefill on gfx950 (1.73 to 2.09x faster prefill kernels; +16.7% geometric-mean throughput at 68K/350 TP8): #35770
- [AMD] Perf Kimi-K3 MoE optimization: #33838
- [AMD] Perf Kimi-K3 fuse ROCm KDA decode boundary: #34198
- [AMD] [Kimi-K3] Fuse the KDA input projection into a single GEMM on ROCm: #35176
- [AMD] support qlen>1 for aiter gluon path for Kimi K3: #37601
- [AMD] Support aiter fa mha chunked kv for Kimi-K3: #37691
- [AMD] Enable topk v2 GLM ROCm (+10.2% throughput per GPU and -9.9% median TPOT geometric mean on MI355X): #36851
- [AMD] Enable Fast Triton Sparse MLA backend: #30575
- [AMD][gfx95] Fill the chunked-prefill compute budget exactly (mean TTFT -5.4 to -9.5% on GLM-5.2-MXFP4 MI355X TP4): #32888
- [ROCm] Take the fused DSA metadata kernels and drop redundant work from the absorb path: #37124
- [ROCm] Make DSA indexer top-k exact with cooperative selection (about 3x faster top-k kernel): #37591
- [AMD] Skip unused TOPK v2 plan kernel on ROCm: #37580
- [AMD][GLM-5.2] Keep GlmMoeDsa MoE e_score_correction_bias in fp32: #37133
- [AMD] Enable FP4 indexer for Deepseek V4 (+5.8% throughput per GPU, -5.5% median TTFT on MI355X): #37353
- [AMD] Fix FP4 indexer OOR: #37660
- [AMD][DSv4] Fuse the DSv4 FP4 indexer prefill-schedule preamble into one kernel (output throughput +15.3% at concurrency 4): #37764
- [AMD][DSv4] Switch output projection gemm (oproj_a) to fp8: #37423
- [AMD][DSv4] Fuse inverse-RoPE into the fp8 wo_a quant (stacked on #37423) (decode TPOT -14.3% vs bf16 at 128K/1K, concurrency 64): #37658
- [AMD][DSV4] feat: enable fp8 two-pool unified_kv on gfx950 (about 1.5x more resident KV tokens): #37413
- [AMD][DSV4] Reland unified-KV pool sizing and SWA ring accounting, fully gated (+83.6% full-attention KV tokens): #38192
- [AMD][DSV4] Skip the paged SWA page return under the per-request ring: #38571
- [AMD][gfx95] DSV4 wo_b (dp-attention): route to tuned bpreshuffle GEMM instead of triton: #38227
- [AMD] Fix DSV4 unified attention sink TP slice: #35092
- [AMD] Fix Dspark accept length and reduce host bubble on DSV4: #39116
- [AMD] Fix DeepSeek block-FP8 loading on gfx94x: #38446
- [AMD] gfx950 assembly attention for EAGLE verify, draft extend and decode (median TPOT -25.6% on Qwen3.5-397B MXFP4 TP4): #37465
- [AMD] gfx950 assembly attention: length-aware split-KV for dynamic workload: #39172
- [AMD] Parallelize aiter spec-decode KV index building over token blocks: #37659
- [AMD] Quantize the bf16 MTP draft experts online to MXFP4 for Qwen3.5 (+13.6% throughput per GPU, -10.3% median TPOT): #38748
- [AMD][Fix] Fix aiter bpreshuffle GEMM for output sizes it cannot dispatch for qwen3.5 mxfp-attn-fp8-v2 TP4: #37564
- [AMD][DCP 1/N] add dcp support for aiter backend: #34432
- [AMD] aiter: honor per-layer softmax scale: #38754
- [AMD] aiter: fail loudly on cross-layer KV sharing in target_verify: #38755
- [AMD] aiter: resolve SWA KV pool for draft workers + guard paged decode: #38756
- [AMD] aiter: route head_dim>256 prefill through Triton unified_attention: #38757
- [AMD] Allow aiter attention backend for Gemma-4 (about 17% more output throughput than Triton at concurrency 128): #38758
- [AMD] Skip AITER FP8 ASM prefill when GQA is unsupported: #38467
- [AMD] fix: use the hardware fp8 e4m3 convert on gfx950: #37140
- [AMD] Fix weight checking for AITER-shuffled block FP8 weights: #34330
- [AMD] Fix Quark load of MiniMax-M3 MXFP4 index_qkv_proj: #37254
- [AMD] Restore SWA reprefill-tail on UnifiedRadixCache when HiCache is off: #32759
- [AMD] Fix registered HiCache host pointer aliases: #35233
- [ROCm] Raise HiCache JIT block quota for mapped-host throughput: #39036
- [AMD] Fix EAGLE crash when no kv_index_translator is bound on the DSA fp8 read door: #38318
- [AMD] CI: fix Lean decode crash on the EAGLE path: #37119
- [MoE][ROCm] Admit the unified Triton router on ROCm, including single-group routing: #38328
- [AMD][Diffusion] Migrate FlyDSL fused norm kernels to the v0.3.0 stable API: #36349
- [AMD] Fix ROCm VAE Conv2D fast path breaking spatial-parallel decode: #34424
- [AMD][Fix] Fix regression in jit build error with FLUX.2-dev on gfx1250: #38329
- [AMD] Add gfx1151 (Strix Halo / Ryzen AI MAX+) Docker image: #33939
- [AMD][CI] Publish ROCm 10 release images and kernel wheel: #38763
NPU / Ascend
- [NPU] Add NPU arch35 support and enhance DSV4 processing in DeepSeek-V4: #37373
- [NPU] Support DSV4 host memory cache management: #37382
- [NPU] Add sparsity-driven KV offload for DeepSeek DSA on Ascend (decode throughput 1.6x): #33089
- [NPU] Support GLM5.2 and FP8 DSA&Indexer kvcache for 950: #38250
- [NPU] glm5.2 fp8 memory opt: #38807
- [NPU] Adapt DFlash2 speculative decoding to Ascend NPUs: #35629
- [NPU] Support DFlash speculative decoding for MiMo-V2.5-Pro (mxfp4): #37565
- [NPU] Enable non-greedy MTP sampling: #32495
- [NPU] Support EAGLE when PP enabled in prefill nodes: #32207
- [NPU][HiCache] Optimize HiCache L2 IO with Memfabric acc_offload: #38826
- [NPU][HiCache] Add Ascend Memcache Hicache L3 storage backend: #38827
- [NPU] support mf device urma and host rdma trans type: #38174
- [NPU] Strip padding before FIA kernel for vision encoder padded sequences: #36329
- [NPU] Fix ViT graph key layout handling: #37758
- [NPU] fix pp 2 hang on npu: #38249
- [NPU] [Fix] Fix NPU MLA HiCache backup accessing missing data_ptrs.: #36813
- [NPU] fix extend_seq_lens_cpu shape in eager mode: #36843
- [NPU] Fix error due to missing parameter quant_linear passing: #37384
- [NPU] Fix device mismatch in SWA mask for DSpark verify graph capture: #39353
- [NPU][Fix] Disable pinned memory to fix DeepSeek-V2 DP-attention hang: #39423
- [Feat] Add base NpuSRTPlatform implementation: #36472
- [Diffusion] [NPU] Optimize LTX-2/2.3 inference performance for NPU (latency -7.6 to -10.9%): #34722
CPU / Intel / XPU
- [CPU] Support FP8 KV cache: #32733
- [CPU] Implement fused QK Norm and RoPE kernels: #37748
- [CPU] Add native CPU kernel for MurmurHash32: #35604
- [CPU] Support Qwen3.8 text+video: adding torchcodec, ffmpeg and removing pin_memory: #35492
- [Diffusion][CPU] Enable MiniMax-H3 on Xeon CPU: #35147
- [CPU] Fix shm allreduce collision and sglang-router import: #37179
- [CPU] Fix native KV hash compilation in Xeon image: #34767
- [CPU] Update base image to Ubuntu 26.04: #35313
- XPU: Enable GLM5.1 (GlmMoeDsaForCausalLM) DSA Attention: #24959
- [Intel GPU] Add MLA support to Intel XPU Attention backend for Prefill: #35866
- Speculative Decoding support for intel_xpu attention backend on XPU target: #30548
- DFLASH support added for XPU: #32798
- [XPU] Support GPT-OSS MXFP4 checkpoints on Intel XPU: #35751
- [Feature][Intel XPU] Add memory saver support for Intel XPU via upstream torch_memory_saver: #29935
- [Intel][XPU] Add NUMA node binding support for Intel XPU: #31113
- [XPU][Diffusion] Enable MiniMax H3 on XPU platforms: #33366
- [XPU] Re-add intel xpu on triton paths in diffusion platforms: #36654
- [XPU] Adapt device agnostic API usage: #32093
- [XPU] Make checkpoint_engine worker device-agnostic: #32382
- [XPU][Fix] Pack device-pointer tables as uint64 to avoid 64-bit address overflow: #35051
- [XPU] Add xpu forward in Gemma3RMSNorm & Add test and benchmark for Gemma3RMSNorm: #36278
- Fix/whisper xpu varlen encoder decoder: #36298
- Xpu/weekly simple model enablement 2026 08 30: #37193
- [Intel GPU] Align XPU toml file for rust support: #31031
MUSA / Moore Threads
First installation path for Moore Threads GPUs: a MUSA Dockerfile, install guide, and runtime extras.
- [MUSA] Add installation guide and Dockerfile: #36709
Security
- [Rust] Bound multimodal media ingress: #37967
- [Router] Raise chat body cap to 32 MB for multimodal payloads: #38735
Cookbook Updates
- MiniCPM5-2B deployment guide: link
- DeepSeek-V4: DGX Spark (2x GB10) Flash FP4 recipes, Flash Vision verified on B200 and GB300, Pro B200 FP4 agentic HiCache with DSpark: #37479, #37737, #37468, #37492, #38026
- DeepSeek-V4 on MI355X: agentic-workload arguments, DSpark config, and PD disaggregation recipes for all three strategies: #38677, #39252, #39396
- GLM-5.3-Flash: FP8 speed data on B300, H100, and B200, NVFP4 rows, MTP with the flashinfer_trtllm runner on Blackwell, and a breakable prefill CUDA graph opt-in: #37412, #37576, #39213, #38522
- GLM-5.3: DFlash2 speculative option: #37392
- GLM-5.2: NVFP4 on B200/B300 for AgentX with HiCache, and MI355X MXFP4 on the Triton DSA backend with TP8/EP1 and TP4/EP4 recipes: #35368, #39106, #39230, #39406
- Kimi-K3: measured B300 1x8 numbers, DCP under HiCache L1+L2 with DSpark, and measured MI350X/MI355X cells: #37878, #39190, #39029, #37781
- Qwen3.8-Flash-Next NVFP4 on DGX Spark (1x, 2x) and RTX PRO 6000, and the NVIDIA NVFP4 export for Qwen3.8-27B: #37995, #38611
- Qwen3.5: FlashInfer GDN on B200, FP8 on B200/B300 with trtllm-gen MoE and symmetric memory, and MI355X MXFP4 with the AttnFP8-V2 checkpoint and HiCache: #37360, #38374, #39104, #39358
- MiniMax-H3: GB300 and GB200 recipes, DGX Spark, a measured RTX 5090 desktop recipe, and VDN-H3 numbers on B200: #38296, #37456, #39373, #39244
- JoyEcho: measured H200 residency and BCG recipe: #38534
- Diffusion performance guide: per-model tuning decision table and single-GPU B300 notes: #38148, #37891
Dependencies
- [Deps] Retire the CUDA 12 lane: #38404
- chore: bump sglang-kernel version to 0.4.7: #39346
- chore: bump sgl-kernel version to 0.4.7: #39324
- bumping sgl-deep-gemm to 0.2.0: #39371
- [Deps] Raise smg-grpc-servicer floor to >=0.9.0 to unbreak SMG E2E CI: #38801
- [CI] Install sgl-eval from PyPI through the test extra: #37504
- Add missing test dependencies to pyproject.toml variants: #36098
- [DeepSeek-V4.1] Bump FlashMLA to the fork's rebase head (v4.1 kernels): #39171
- [AMD] Update ROCm AITER pin to 4ad9983: #37784
- [AMD] [Docker] Update MoRI to v1.2.3: #38411
- [AMD][CI] Retire the ROCm 7.0 kernel wheel: #38767
- [NPU] Update sgl-kernel-npu version to 2026.9.0 and move memfabric deps into pyproject: #37399
- [NPU] Bump memfabric and sgl-kernel-npu versions in docs and pyproject_npu.toml: #38437
- [XPU] install xpu-kernel by released wheel: #37394
- [XPU] Add Regular Docker Image Release workflow for Intel XPU: #37340
- docker(xpu): unblock nightly build (setvars.sh + sgl-kernel rename): #38617
- pyproject(xpu): drop human-eval git dep to unblock image build: #38824
Breaking Changes & Upgrade Notes
- The CUDA 12 lane is retired. No
-cu129/-cu12wheels or images are published from this release on; already-published tags are unaffected, and v0.5.19 is the last release with a CUDA 12 lane: #38404 - Prefill context parallelism v1 is removed. The strategy-based CP path is canonical; the
SGLANG_ENABLE_CP_V2gate, the v1 runtime, and the generic v1 CLI options are gone, and CP APIs are renamed to versionless names: #36223, #36228, #36229.--enable-prefill-cpis rejected on HIP, NPU, and MUSA until their CP refactor lands; decode CP and non-CP inference are unaffected: #38293 /v1/responsesstorage is off by default. Start with--enable-response-storeto keep retrieval,previous_response_idchaining, and background requests; PD deployments cannot enable it and reject those features at admission.background=truewithstore=falseis rejected: #39122- Sampling-mask capacity moves to
--sampling-mask-max-tokens(default 4096).SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENSis rejected at startup, and masks under PD requireSGLANG_ENABLE_DISAGG_SAMPLING_MASK=1on both prefill and decode servers: #36631 - The
cutlass_mlaattention backend, the non-Marlingptqquantization choice, the AWQ AOT dequantize kernel, and Dual Chunk Flash Attention are deleted. GPTQ checkpoints continue throughgptq_marlin;auto-roundandmoe_wna16lose their non-Marlin GPU fallback: #32114 - Seventeen deprecated CLI flags are removed and
get_global_server_argsnow raises; read resolved values from the namespace bags or the raw input fromget_server_args(): #38375. The config tier, includingServerArgs, is nowmsgspec.Struct, sodataclasses.asdictand dataclass introspection on it no longer work: #38753, #38958 --dcp-comm-backendnow resolves per platform for every model:fi_a2aon Blackwell when the DCP group sits inside one node or MNNVL domain, otherwisea2aon CUDA and ROCm; pass--dcp-comm-backend ag_rsto restore the old default: #37767, #39165- SM120 FP8 linear layers with per-channel weight scales take the per-tensor route (cudnn/nvjet) instead of channelwise CUTLASS: #38006. On 32 GB GPUs the decode CUDA-graph cap rises from 24 to 48 (160 at TP >= 4) and chunked prefill from 2K to 4K: #37898
- An undersized SWA pool now fails at startup with a
ValueErrorinstead of hanging at warmup: #37610 - gRPC clients:
GetIsReadyis replaced by theWatchEngineStatestream, which publishes health, pause state, instance ID, and discovery snapshots: #39915 - Diffusion defaults change: video DiTs stay resident on GPUs with enough free memory unless
--dit-cpu-offloadis passed explicitly (#38001); the SDPA attention backend is chosen by measurement, retiring the no-opallow_cudnn_sdp(#38689); component loader capability switches are replaced by resolver contracts that fail closed (#36824) - ROCm 10 is the default AMD lane. ROCm 7.0 CI, nightly images, and the
rocm700kernel wheel are retired; ROCm 10 release images ship for MI30x and MI35x with a+rocm1000kernel wheel, and ROCm 7.2.x images remain: #38659, #38632, #38767, #38763 - The CPU image moves to Ubuntu 26.04 with GCC 15.2 as the default compiler: #35313
- Contributors: Python formatting moves from black to ruff-format, with a
.git-blame-ignore-revsentry for the reformat commit: #37210, #37695
New Contributors
- @EricKing626 made their first contribution in #36851
- @hekhong-png made their first contribution in #34362
- @Ankur-singh made their first contribution in https://github.com/sgl-project/sglang/pull/37360
- @hn1209 made their first contribution in #32218
- @akashpalla made their first contribution in #36220
- @Viol2000 made their first contribution in #37300
- @wes-lyu made their first contribution in #35443
- @akhauriyash made their first contribution in https://github.com/sgl-project/sglang/pull/37655
- @dujifeng made their first contribution in #37616
- @tristan-me made their first contribution in #37332
- @Bo-Vincent made their first contribution in #37731
- @hickeyma made their first contribution in https://github.com/sgl-project/sglang/pull/37788
- @kediwu0331 made their first contribution in #36380
- @littlefatfat made their first contribution in #33824
- @heachary made their first contribution in #37764
- @mottopanikeiku made their first contribution in https://github.com/sgl-project/sglang/pull/32172
- @Grypse made their first contribution in #37965
- @amykchang made their first contribution in #34488
- @DUXin-ece made their first contribution in #37971
- @Beihao-Zhou made their first contribution in #32405
- @zqlcode made their first contribution in #32151
- @yuttian1 made their first contribution in https://github.com/sgl-project/sglang/pull/30315
- @yu-depend made their first contribution in #38171
- @sghhhh made their first contribution in #37199
- @JohnQinAMD made their first contribution in #37720
- @samnordmann made their first contribution in #30430
- @kms07 made their first contribution in #36654
- @lixiufei-leo made their first contribution in #38227
- @pinkgom made their first contribution in #38116
- @jeffreywang88 made their first contribution in #31415
- @nogumanov made their first contribution in #37143
- @kedarpotdar-nv made their first contribution in https://github.com/sgl-project/sglang/pull/38374
- @EdwardXuy made their first contribution in https://github.com/sgl-project/sglang/pull/38125
- @cl-vv-h made their first contribution in #32495
- @AnuSajikumar6264 made their first contribution in #30345
- @sagearc made their first contribution in #36234
- @davidli1515 made their first contribution in https://github.com/sgl-project/sglang/pull/38399
- @vorapolsiloai made their first contribution in #38467
- @123liuziming made their first contribution in #37789
- @nzbej made their first contribution in #33089
- @jianuo-huang made their first contribution in #36899
- @LinPoly made their first contribution in #36655
- @rystewart-nvidia made their first contribution in #35599
- @hhy-seven made their first contribution in #35486
- @vpremseth23 made their first contribution in #38754
- @MeowMiaoJ made their first contribution in #36278
- @rishabhsinha17 made their first contribution in https://github.com/sgl-project/sglang/pull/34977
- @yousafshah made their first contribution in #38693
- @Xinyu-Kang made their first contribution in #34330
- @simple-sun made their first contribution in #37818
- @tnguyeng made their first contribution in #37254
- @knlb-tml made their first contribution in #38908
- @desmond-intel made their first contribution in #34556
- @jlqibm made their first contribution in #37679
- @littledgg made their first contribution in #38486
- @Hritik003 made their first contribution in https://github.com/sgl-project/sglang/pull/37015
- @200lz made their first contribution in #36141
- @cx22757 made their first contribution in #37382
- @ichbinblau made their first contribution in https://github.com/sgl-project/sglang/pull/39396
- @CharlesXu-HQ made their first contribution in https://github.com/sgl-project/sglang/pull/38097
Highlights
786 PRs from 214 contributors.
New models in this release (see the cookbook for all supported models):
| Model | Type | PRs | Cookbook |
|---|---|---|---|
| Qwen3.8 (2.4T-A95B) | Autoregressive | #35758 ⭐ | link |
| Qwen3.8-27B | Autoregressive | #34859 | link |
| dots3.note | Autoregressive | #33829 ⭐ | link |
| Ling-3.0-flash | Autoregressive | #33561 | link |
| Ling-3.0-tiny | Autoregressive | #33561 | link |
| Spark2.5 | Autoregressive | #35963 ⭐ | |
| MiniCPM-SALA | Autoregressive | #30360 | |
| Granite 4.2 | Autoregressive | #36286 | link |
| LongCat-Image-Edit & Edit-Turbo | Diffusion | #35829 |
Cookbook updates:
- GLM-5.3 deployment guide: link
- PaddleOCR-VL deployment guide: link
- Kimi-K3 on Ascend A3: #35508
- Kimi-K2.7-Code-MXFP4 on MI355X: #36246
- Qwen3.5 MXFP4 on MI355X with an FP8 KV cache or a HiCache host-memory tier: #35445, #36245
- MiniMax-H3 on a 24 GB GPU or DGX Spark, with a consumer-GPU tuning guide: #36169, #35816
- Ling-3.0-flash on DGX Spark: #36364
- Qwen3.8-27B on RTX 5090, RTX PRO 6000, and DGX Spark, re-measured: #35825
Beam search. SGLang can now do beam search. Pass beam_width in your request and you get back the n best sequences instead of a single sample. It works out of the box next to regular requests, though it does not yet mix with speculative decoding, disaggregation, DP attention, or HiCache (#31626).
DeepEP v2. DeepEP's new ElasticBuffer engine is available as --moe-a2a-backend deepep_v2 for DeepSeek-V3/V4 and Qwen3-MoE in FP8. Its buffers have a fixed size, so decode can run under CUDA graphs even across nodes. Performance is on par with the classic backend (#35634, #34923).
LayerNorm sequence parallelism. With --enable-layernorm-sp, each tensor-parallel rank normalizes only its own share of the prefill tokens instead of all of them. That takes 3.5% off Qwen3-8B prefill on H100 and 5.6% on B200, and the saving grows with the TP degree. Dense Qwen3 models only for now (#30915).
W4A8 MoE on Hopper. If you serve MXFP4 experts on Hopper, you can now quantize the activations to FP8 as well with --flashinfer-mxfp4-moe-precision fp8. DeepSeek-V4-Flash gains about 12% output throughput with no change in GSM8K accuracy. Needs FlashInfer 0.6.18 (#34967).
DCP on the default Blackwell MLA backend. Decode context parallelism now runs on trtllm_mla, not just CuTe DSL and Tokenspeed. It pays off at long context: at 128K input, plain TP stops scaling around 680 tokens per second on eight B200s, while DCP keeps going as concurrency grows (#33926).
Faster speculative kernels. DSA prefill top-k moves to the v2 kernel, 1.3 to 1.8 times faster on B200 (#35175). KDA models get an opt-in fused accept path, SGLANG_OPT_KDA_FUSED_ACCEPT_STATE=1, that cuts MTP verify-and-commit time by 45% to 63% on Kimi-Linear shapes with bit-identical output (#33722).
Unified radix tree by default. The unified tree is now the cache for every model, not just hybrid ones (#35081, see Breaking Changes). It also picked up three things this cycle: PD decode workers can reuse cached prefixes for SWA hybrid models like gpt-oss (#27770), you can attach or detach L3 storage on a running server (#35269), and pipeline parallelism with HiCache L3 stays consistent across ranks (#27010).
Lean attention on AMD. Long or uneven decode batches used to leave many compute units idle on MI300X and MI355X. The new persistent Lean kernel spreads the work across all of them, for up to 1.52x more throughput and up to 3.62x lower inter-token latency on MI355X. It turns on by itself where it helps, and SGLANG_DISABLE_LEAN_ATTENTION=1 turns it off (#33576).
DSA models on ROCm. Disaggregated GLM-5.2 serving now uses the fused top-k seed remap, which brings decode TPOT from 23 ms down to 8 ms on eight MI355Xs (#36714). DeepSeek-V4 gets the v2 top-k kernel, up to three times faster (#36684), and a shared-experts gate fix gives GLM-5.2 up to 16% better TPOT (#36124).
Dependencies. FlashInfer moves to 0.6.18 (#36954), sgl-deep-ep to 0.1.2 (#35450), sgl-deep-gemm to 0.1.7 (#37279), and mooncake to 0.3.13 (#36493). There is a new CUDA 13.4 preview image for Rubin (#36233) and new ROCm 10 images for gfx942, gfx950, and gfx1250 (#36434, #36871).
Full release notes by category below; breaking changes and known issues are at the end.
Rust Server
- [Feature] Add process-local in-memory KV indexer and Router integration: #33370
- [Rust Server] Add e2e latency metadata and fix Sarashina import: #35125
- [Rust] Derive server address and accept signed env values: #37221
- [Rust] Keep sampling and scheduler wire schemas in sync: #37222
- Add configurable HTTP/2 connection window: #36920
- fix(gateway): bump wfaas to 1.0.2 so ContinueNextStep unblocks dependents: #37249
Speculative Decoding
- [KDA] Fused-accept state advance for FlashInfer KDA MTP verify: #33722 ⭐
- [Spec] DFlash2: local convolution + candidate selector (3.43x over no-spec at batch 1 and about 24% over DFlash at concurrency 64): #35371
- [Spec] Support quantized target lm_head in the DFlash2 selector: #35496
- [Spec] Add LFM2 and LFM2-MoE DSpark speculative decoding support (1.05 to 2.42x faster decoding on LFM2.5 targets, 1xH100): #31041
- [Model] Support Nemotron 3.5 Lightning speculative decoding (GSM8K 94.6% to 95.8% across MTP, DFlash, and DSpark on GB300): #36186
- [Spec][LoRA] Support multi-adapter LoRA with EAGLE/NEXTN/DFLASH/DSPARK speculative decoding: #34337
- [Spec][DSA] Add --speculative-dsa-topk-backend: #36313
- [2/N][Mixed] Mixed chunk prefill with spec enabled: #36933
- Support custom draft worker classes in DSpark: #35397
- Make draft attention backends extensible: #35932
- [Memory] Borrow CUDA graph pool storage for EAGLE sampling: #35375
- [Spec] Fix Dspark and Dflash state divergence across TP rank: #33614
- [Fix][Spec] fix startup crash and reduce CUDA graph memory usage for speculative adaptive: #35275
- Fix DSV4 DSpark sample-from-anchor initialization: #36419
- [Fix] Drop the duplicated DSpark draft sample_block call: #36934
- fix(gemma4): quantize MTP bridge projections (Gemma-4 FP8 MTP acceptance from 0% to 60%): #32440
Piecewise & Breakable CUDA Graph
- [Feature] Support PP in full prefill CUDA graphs: #35451
- [Feature] Coordinate FullCG prefill across DP-attention ranks: #35640
- [PP] Support prefill CUDA graph proxy tensors (PP prefill up to 2.48x at 2K-token forwards on Qwen3.5-397B, GB300): #36248
- [Performance] Reduce idle DP work in breakable prefill CUDA graphs (median TTFT -18.9%, QPS +12.9%): #33871
- [Cohere Command-A-Plus] Optimize decode and BCG capture on SM10X (decode TPOT 2.28x faster at batch 1, up to 9,152 tok/s on 4xGB300): #36624
- [Fix] Fix full prefill CUDA graph padding and EAGLE capture: #35588
- Converge the two SWA predicates, and stop conditioning the capture sink on the pool: #37550
Attention Backends
- [DCP] Support decode context parallelism on the trtllm_mla decode path: #33926 ⭐
- [DSA] Route the ragged prefill top-k to the v2 kernel: #35175 ⭐
- [DSA] Trim top-k v2 output modes and tighten its PDL waits: #35041
- Add fmha_v2 attention backend for SM90/120 (about 15% faster than FA3 at the kernel level): #23112
- [FlashInfer 0.6.18] feat(dsv4): support --dsa-topk-backend flashinfer with fused top-k: #33237
- [DeepSeek-V4] Add Q8KV8 sparse MLA prefill runtime backend (prefill throughput +4.4% to +7.5% from 3.5K to 160K input on H20): #32327
- Split TRTLLM MHA decode batches by KV sequence length (mixed-length decode step -10.2% on GB300): #34888
- [Performance] Tune FlashInfer EXTEND for DP prefill (DP prefill throughput +4.9% to +6.8%): #36219
- [Triton] Bound the sliding-window extend-attention KV loop: -86.6% on SWA layers, -9.4% prefill GPU, bit-identical: #34462
- [Kernel] Skip reserved writes in MLA KV cache: #36003
- [Fix] Skip padded state slots in the chunked GDN kernel: #33431
- [Fix] Fix Nemotron-H Mamba illegal memory access under DP attention with CUDA graph: #34561
MoE & Expert Parallelism
- [Feature] Add DeepEPv2 (ElasticBuffer) MoE A2A backend: #35634 ⭐
- Apply latest DeepEP branch: #34923 ⭐
- [MoE] Add FlashInfer SM90 MXFP4 W4A8 CUTLASS MoE (W4A8 kernel 1.63 to 2.08x faster than W4A16, +11.7% output throughput on DeepSeek-V4-Flash): #34967 ⭐
- [DeepSeek V4] Add W4A4 MegaMoE server flag: #35918
- [DeepSeek V4] Default FP4 checkpoints to FlashInfer MXFP4 MoE: #35919
- [DeepSeek-V4] Enable shared-experts fusion on the flashinfer_mxfp4 (trtllm-gen) MoE path (TTFT -13% to -21%, warmed GSM8K e2e +5-7%): #35505
- Support deepseek v4 and kimi k3 on ssd (DeepSeek-V4-Flash prefill 2.24x and decode 1.49x faster than llama.cpp on one 5090): #35314
- perf: overlap Qwen shared expert with DeepEP routed experts (+2.1% output throughput on 24xGB300 disaggregated Qwen3.5-397B): #34938
- feat(humming): FP8 DeepEP dispatch for humming MoE backend (TPOT -35% and TTFT -23% vs BF16 dispatch on H20): #31429
- feat(humming): support native W4AFP8 checkpoint schemas (TPOT -22% to -29%, TTFT -10% to -11% on H20): #32033
- [MoE] Make simulated expert routing support DP>1, and fuse into one triton kernel: #29718
- [Blackwell] Reserve SMs for DeepGEMM MegaMoE grid barriers: #36657
- [DeepGEMM] Deduplicate JIT precompile across local ranks: #36231
- [Perf] Tune the W4AFP8 DeepEP low-latency requant launch geometry (requant kernel up to 10.6x faster at 256 experts): #35760
- [Kimi K3] Select FlashInfer MXFP4 for SM107 auto MoE: #35554
- [Fix] Route the Mooncake MoE A2A backend through Kimi K3's EP-A2A / SP-MoE fast path: #36862
- Laguna: config-driven MoE router scoring: #35362
- Sync FlashInfer autotune tactic choice across TP ranks: #35343
- Fix OOB read in mxfp4 MoE weight scales on Hopper: #36456
- Fix stale GLM MoE routing after runtime weight updates: #35883
Quantization
- [FlashInfer 0.6.18] add FlashInfer CuTe DSL NVFP4 W4A16 mode (Qwen3-30B-A3B W4A16 GSM8K 0.965 to 0.980, 1,662 to 1,858 tok/s): #35120
- [Quant] Load compressed-tensors quantized lm_head instead of value-casting it: #35228
- [Quant] Load compressed-tensors kv_cache_scheme scales: #35455
- Support mxfp8 KV cache in PD transfer: #35718
- Support CPU offload for mxfp8 KV cache: #35888
- [Kernel] Fix SM90 FP8 decode regression with benchmarked M/K/N routing (restores about 215 tok/s decode from 190 after the SM90 FP8 routing regression): #37018
- [NVIDIA] Fix SM107 MXFP8 activation prep: #35405
- Fix GPU kernel ordering and MXFP8 quantization dispatch: #37331
- fix(kimi-k3): preserve dense ModelSlim MLA weights: #36603
- Support Intern-S2-Mobius FP8: #34908
Parallelism & Disaggregation
- [Feature] Megatron LayerNorm sequence parallelism (--enable-layernorm-sp): #30915 ⭐
- [PD] Decode-side radix cache for SWA hybrid models (unified radix tree): #27770 ⭐
- [Refactor] New EPD: #30398
- [PD] Pack DCP1→DCP-N PD KV transfers into dest-contiguous RDMA blocks (PD transfer time -28% to -39% on Kimi-Linear-48B, 8xB300): #35762
- [PD] Deferred decode-side KV release for aborts mid-transfer: #35049
- [PD] Deferred decode-side KV release for the NIXL backend: #35360
- [PD] Overlap prefill DP-rank bootstrap queries: #35071
- [PD] Fix MORI-IO ABORT bootstrap message handling: #29133
- [Fix] Assert the page-aligned SWA evict floor on both PD decode prealloc paths: #35396
- fix(disagg): refresh stale prefill bootstrap metadata: #36029
- [Fix] DP attention: correct the decode->extend prefix off-by-one (DP attention GSM8K back from 70.4 to 94.2 on Qwen3-30B-A3B): #37505
- [Fix] Select custom all-reduce v2 by topology capability: #35061
- [Kernel] Split the custom all-reduce communicator into push/pull planes: #35735
- TP/PP Consensus checker: #34406
- [CP V1 Deprecation 1/5] Migrate tests to strategy-based prefill CP: #36222
Scheduler & Runtime
- [Feature] Beam search support: #31626 ⭐
- [1/N][Mix] Mixed Chunk Prefill Base: #36288
- [Scheduler] Add configurable decode interval after prefill: #35017
- Publish per-scheduler load on a dedicated socket for load-aware routers: #34608
- [Fix] Fix min-new-token EOS handling: #31378
- fix: preserve output logprobs without input logprobs: #34627
- [Sampling] Restore finite top-k requirement for sampling masks: #35205
- [Sampling] Fix int32 offset overflow in top-k renorm Triton kernels: #35571
- Handle unlimited tokenizer context lengths: #37054
- [Constrained] Support MistralCommon tokenizers in the XGrammar backend: #35215
- [OpenAI] Drop empty assistant turns for mistral_common tokenizers: #35915
- [Fix] Pass Anthropic thinking history as reasoning_content for custom chat encoders: #35480
- [Fix] Resolve tool argument types through top-level anyOf/oneOf/allOf: #36626
- feat(openai): Accept the input_audio content part in chat completions: #33606
- Make the scheduler track the published weight version: #35925
- Report per-token weight-version spans in generation meta info: #35926
- [FEAT] Weight Daemon abstraction: #33279
- [Weight Cache] Support static DP/EP layouts: #33684
- [Fix] Account resident weight memory in KV sizing: #34053
- Fix KV cache pool sized far too small when weight-loading memory is still referenced: #36583
- Profiling Enhancements [2/3]: detailed execution step annotations: #24911
HiCache & Radix Cache
- Using unified radix tree by default for all case: #35081
- [Unified Tree] feat: support runtime attach/detach for historage: #35269 ⭐
- [HiCache] Fix PP inconsistency with HiCache L3 (#22607): #27010 ⭐
- [Radix Cache] Add Rust TreeCore backend with shared parity tests: #32710
- [Unified Cache][1/N]: Support cache contract for external linker: #37091
- [Unified Cache][4/N]: Add Mooncake backend for external linker: #37205
- [HiCache] Buffer-only mode for HiCache host memory layer: #34798
- [HiCache] buffer mode: decide staged-fetch fate against the live tree: #36834
- Fix buffer-mode HiCache load-back ownership races; add optional prefetch anchor lock: #35769
- [HiCache] Batch PP write and load completion sync: #33473
- [HiCache] Stop populating host-pool mmaps twice (-13% allocation time): #36705
- [HiCache] Retry L3 storage prefetch after a missed attempt: #36227
- [HiSparse] Support hisparse multi-step swap io kernel (batch-64 swap latency -48.2% on H20): #32162
- feat(unified-memory): dense KV views for uniform-row MHA/SWA models: #34602
- feat(unified-memory): read unified pool from attention backends fa3/flashinfer/trtllm_mha/flashmla: #34613
- feat(unified-memory): three sub-pools for mamba + hybrid-SWA models: #35177
- feat(unified-memory): byte-budget sizing, feasibility floor, and a conservation verifier: #35158
- [Unified Memory] Stop eviction when shared allocation capacity is sufficient (cache survival 35.7% to 67.9%, replay E2E -37.9%): #33091
Multimodal
- [Perf] PaddleOCR-VL: overlap page preprocessing, pack the ViT, enable prefill CUDA graph (1080p page TTFT 235 ms to 120 ms, 1.84x req/s on H200): #35318
- [VLM] Route every multimodal processor through the worker pool's call site: #35342
- [VLM] Split Pixtral multi-image features before the CUDA IPC wrap: #35463
- Scatter mm embeddings with row index_copy_ instead of masked_scatter_ to cut transient GPU memory: #37070
- [Fix] Re-encode multimodal embeddings after cache mismatch: #36595
- [Fix][VLM] keep Qwen3-VL MoE inference deepstack order: #34690
- [Kimi-K3] Fix "wrong grids" crash in DP-sharded vision preprocessing: #35305
- Fix transcription & audio-understanding for ASR/audio/speech models: #32611
Model Support & Optimizations
- Add SGLang Granite SWA support via existing Granite models: #35794
- [Fix] Transformers-fallback (GPT-NeoX) + KV pool config (DeepSeek-VL2): #35244
- Fix BailingMoeV3 reading enable_dp_lm_head off live topology instead of config: #36584
- [k3] declare packed_modules_mapping on
KimiK3ForConditionalGeneration: #36211 - [Fix][Qwen3.8] Support Qwen3.8-MXFP4 DCP by registing Qwen3_5 text-only archs in mamba radix cache whitelists: #35297
- Fix DeepSeek V4 multistream QKV buffer lifetime: #36547
Kernel Library
- [Kernel] Add H200 MoE configs for Qwen3.5 and Qwen3.6 (+3.8% throughput on Qwen3.5 FP8 and +7.1% on Qwen3.6 FP8, H200): #35374
- [Kernel] Add GB300 Triton MoE configs for GLM-4.5 FP8 (decode throughput +21.9% on GB300): #37159
- [Kernel] Raise shape limits in shared FLA and MoE kernels (ported from #36507): #37317
- [Kernel] Declare the PyTorch ABI dependency in sglang-kernel wheels: #36465
Configuration System
- [Core] Refactor server argument choices: #36586
- config: constructing a config no longer resolves it: #35907
- config: ServerArgs holds the raw input: #36255
- config: resolution declares, and nothing writes a field: #36618
- [Config] Round 5.1: the published-side readers ask the bags, and a platform fact gets one address: #37086
- [Config] Round 5.2: the per-model declarations get their own modules: #37087
SGLang-Diffusion
- [Diffusion] Optimizing MiniMax-H3 for consumer-level GPUs: INT8 Linear + pluggable DiT attention backends (MiniMax-H3 on one 24 GB GPU, denoise 1.34x faster with exact attention and up to 2.48x with approximate attention): #34581
- [Diffusion] feat: load GGUF transformer checkpoints (MiniMax-H3): #35370
- [Diffusion] Support MiniMax-H3 pruned safetensors checkpoints: #35418
- [Diffusion] Load Comfy NVFP4 MiniMax H3 checkpoints: #36044
- [Diffusion] Load MiniMax H3 GGUF text encoders: #36055
- [Diffusion] Load Diffusers MiniMax H3 components natively: #36067
- [Diffusion][MiniMax H3] support subblock sparse attention on SM90: #34680
- [Diffusion][MiniMax H3] Restrict MiniMax-H3 SubBlock sparsity to video queries (1.31 to 1.46x faster with more consistent audio): #35850
- [Diffusion] ComfyUI: add a MiniMax-H3 node and a generic extra-fields passthrough: #35352
- fix(diffusion): MiniMax-H3 dp_size>1 deadlock and cross-request audio determinism: #36398
- [Cosmos3] Add cosmos3 transfer capability: #34747
- [Diffusion] RL rollout support for the Cosmos3 pipeline: #34197
- [Diffusion] Optimize Pi0.5 inference and bounded graph serving (Pi0.5 end to end -9% eager, -68% with bounded graphs on H200): #34599
- [Diffusion] Per-request lossy accelerations: Cache-DiT, CFG gating, attention backend override: #35339
- [Diffusion] Allow Cache-DiT with DiT layerwise offload (Cache-DiT with layerwise offload 1.84x faster at 30.1 dB PSNR on a 24 GB 4090 D): #35858
- [Diffusion] feat: support out-of-tree models and pipelines: #35713
- [Diffusion] Decouple encoder parallelism from the DiT parallel layout: #34713
- [Diffusion] Reuse SRT Qwen vision and text modules: #35006
- [Diffusion] Reuse SRT quantization contracts and MXFP8 kernels: #36063
- [Diffusion] Load serialized Comfy ConvRot INT8 DiTs: #35994
- [Diffusion] Load serialized Comfy W4A8 checkpoints: #36036
- [Diffusion] Add component-scoped quantization overrides: #36035
- [Diffusion] Add composable component weight path CLI: #36078
- [Diffusion] feat: let every layerwise component be configurable: #35688
- [Diffusion] feat: plan pinned host memory against the cgroup cap not the machine (text encoding 2.01x faster on RTX 4090): #35641
- [Diffusion] Enable LongCat breakable CUDA graphs (LongCat-Image denoise -11.9%, e2e -3.4% on B300): #35724
- [Diffusion] Enable SANA-Video breakable CUDA graphs (SANA-Video denoise -13.5%): #35729
- [Diffusion] Accelerate SANA-Video linear attention in quality=high (SANA-Video quality=high e2e -7.2% on B300): #35728
- [Diffusion] Accelerate LingBot Video RMSNorm in quality=high (LingBot-Video quality=high denoise -25.7%, e2e -22%): #35969
- [Diffusion] Fuse SANA-Video interleaved RoPE (SANA-Video e2e -8.2%): #35695
- [Diffusion] Fuse LTX-2.5 decoder 3D RoPE (LTX-2.5 decoder -17%): #35698
- [Diffusion] Fuse LongCat-Image QKNorm and interleaved RoPE (LongCat-Image e2e -17%): #35995
- [Diffusion] Fuse Cosmos3 Nano T2I attention on Hopper (Cosmos3 Nano denoise -12%, e2e -8%): #36571
- [Diffusion][Kernel] Fuse Wan FFN GELU epilogue (Wan2.1 denoise -1.9%): #36592
- [Diffusion][Kernel] Fuse Wan2.2 NVFP4 bias + GELU on Blackwell (Wan2.2 NVFP4 denoise -3.0%): #37075
- [Diffusion] Optimize Qwen-Image TP collectives and attention (Qwen-Image-2512 TP2 e2e -8.6%): #36680
- [Diffusion] Cache Qwen-Image modulation across serial CFG branches: #37090
- [Diffusion] Fuse Qwen-Image FP8 QKV projection and Blackwell epilogue: #37123
- [Diffusion] Fuse Qwen-Image residual norm and NVFP4 quantization (denoise -1.2% on GB300): #37129
- [Diffusion] Fuse FLUX.2 gated residual normalization on Blackwell: #37112
- [Diffusion] Fuse FLUX.2 token concatenation and NVFP4 quantization: #37141
- [Diffusion] Fuse FLUX.2 ModelOpt FP8 producers and QKV packing: #37162
- [Diffusion] Default Hunyuan VAE to tiled decode (Hunyuan e2e +18% and FastHunyuan no longer OOMs in VAE decode on 4xH200): #36012
Local & Desktop AI
- [Multimodal] Fix NVFP4 diffusion models on sm_120 (RTX PRO 6000 / RTX 50xx): #35739
- [SM120] flash_mla: allocate the page-split buffer outside inference mode: #35116
- [MLX] Upgrade to Torch 2.13/MLX 0.32+ and redesign the Torch-MLX tensor bridge: #32984
- [MLX] Fix startup crash when reporting preloaded weights: #37035
AMD / ROCm
- [AMD] Add Work-Centric (Lean) Attention: a persistent-CTA decode kernel for long-context serving (up to 1.52x throughput and 3.62x lower median ITL on MI355X): #33576
- [AMD] support gfx1250 on ROCM 10: #36871
- [AMD] Add ROCm 10 (gfx942 / gfx950) release images: #36434
- [AMD] Enable Mori-EP on kimi-k3 (GSM8K 0.951): #35630
- [AMD][Fix] Route MoRI through the Qwen MoE all-to-all path (Qwen3.5 MoRI GSM8K 0.006 to 0.967): #32039
- [AMD] Enable moe_a2a_backend=mori for DeepSeek-V4 prefill context parallelism: #35611
- [AMD] Add Radix-4 MoE top-k router kernel for Kimi-K3 routing: #34490
- [AMD] Enable aiter mla asm path through padding attn heads for Kimi K3 (GSM8K 0.947 to 0.958): #36356
- [AMD] Improve K3 dspark draft attn kernel perf: #35499
- [AMD] Enable 12-head MLA aiter fp8 Gluon decode (batched bh16bn128).: #34647
- [AMD] DeepSeek-V4: add aiter fused mHC post+pre with cross-layer boundary dispatch: #32577
- [AMD] DeepSeek-V4: route decode wo_a bf16 batched matmul to aiter batched_gemm_bf16: #33313
- [AMD] DeepSeek-V4 MI355X: eliminate bpreshuffle fp8-scale copies at producer sites (MoE down, MLA o_proj bmm): #33166
- [AMD][DSV4] perf: bound the MoRI receive buffer during decode: #36130
- [AMD] Enable deepseek-v4 topk_transform v2 kernel (top-k kernel 2.66 to 2.97x faster, up to 1.07x end to end): #36684
- [AMD] Enable draft_extend CUDA graph for HIP DSA backend: #35672
- [AMD][Spec][PD] Enable the PD DSA fused-TopK seed remap on ROCm (GLM-5.2 PD decode TPOT 23.16 ms to 7.94 ms on 8xMI355X): #36714
- [AMD] Qwen3.5 ASM FMHA chunked-prefill context attention (context-chunk kernel 14.24 ms to 4.04 ms, E2E latency -24.6% to -28.5% at 70K input): #36758
- [AMD] Optimize Qwen3.5 MTP unified attention on gfx950 (+9.6% throughput at concurrency 8 with 70K input): #36330
- [AMD] MiniMax-M3 : Fuse QKV+index proj for block-fp8: #32099
- [AMD] feat(moe): fold padded-topk_ids fill into fused shared-experts append+remap: #31370
- [AMD] [sgl-kernel] Bypass caches for peer traffic in ROCm custom all-reduce: #32832
- [ROCm] Extend the gfx950 extend-attention tile to head_dim <= 128: -43% kernel, -14% TTFT, bit-identical: #34461
- [ROCm] Fix QuickReduce fp16 saturation corrupting bf16 all-reduces (106M non-finite -> 0, +0.3%): #34484
- [ROCm] Define the DSA head-gate graph helpers on HIP: #37118
- [ROCm][Fix] Cap the DSA MQA-logits budget at AITER's buffer_store limit: #36960
- [ROCm][Fix] Use token-level KV indices in the aiter ASM context-prefill gather: #36852
- [AMD][Spec] Fix aiter GQA packing + split-KV routing in NEXTN spec attention (verify & draft_extend) (draft-extend at 88K from ~773 ms to ~5 ms, NEXTN now 1.35 to 1.76x faster than no-spec): #30105
- [AMD] Fix DSv4 draft extend taking the target compression path during prefill: #37713
- [AMD] Fix v4 topk issue: #37439
- [AMD] fix aiter cannot get heuristic kernel regression: #37438
- [AMD] Gate the aiter memory-reserve exemption behind an env var: #37242
- [AMD] Fix Qwen3.5 MTP dropping fused shared-expert weights: #35719
- [AMD] Quark shared-experts gate: recognise a trailing MTP layer (GLM-5.2 median TPOT -16.0% at concurrency 4, -8.4% at 64): #36124
- [AMD] Fix the QuickReduce bf16 cast failing to build for CDNA: #37132
- fix(lora): build the MoE LoRA align JIT kernel on ROCm: #36379
- [AMD] Fix nightly ROCm 7.0 image build: patch missing
<optional>include in AITER topk kernel: #36216 - [AMD] Keep the PTX-inline-asm diffusion norm fusions off on ROCm (fix FLUX warmup crash): #34481
- [AMD] Let the diffusion AITer backend take grouped-query K/V (fix Cosmos3-Nano startup): #34485
- [AMD] [Docker] Upgrade Python 3.12 + torch 2.11 + triton 3.7 in ROCm 7.2.4: #30984
- [AMD] Update ROCm AITER pin to c16d44b: #35810
- [AMD][MORI] Bump MoRI to 7c51d18 for ionic RoCE dmabuf fix (#509): #37286
NPU / Ascend
- [NPU] DeepSeek-V4 adapt sgl-kernel-npu ops (compressor/sparse-attn/sparse-attn-metadata): #35676
- [NPU] Add mxfp4-w4a4 MOE Quantization Support for NPU: #30319
- [NPU] add causal conv1d for ascend kda backend: #35021
- [NPU] [Diffusion] Support MiniMax H3 on Ascend NPU's (Laser attention 1.26 to 1.45x faster than Ascend flash attention): #33569
- [NPU] [Diffusion] support distributed inference pipeline for GLM-Image (14-denoiser disaggregation 3.7x faster than monolithic batching): #31320
- [NPU] [Diffusion] Fix critical Ascend NPU Diffusion regression/bugs & restore 2-NPU CI testcase: #34855
- Support streaming session on NPU: #32597
- [NPU] fix: reach torch>=2.8 CUDA memory-pool APIs lazily via torch._C: #29100
- [NPU][Fix] Fix OOB gather in decode KV allocation when free pool is tight: #35727
- [NPU][Fix] Fix discontinuous input for FIA operator in GLM4.7‑Flash: #36170
- Fix NPUMHATokenToKVPool missing k_data_ptrs/v_data_ptrs: #35820
CPU / Intel / XPU
- [XPU] Support INT4 dense linear (AWQ/GPTQ) for XPU: #30236
- [XPU] Use SYCL kernels for DeepSeek V4 MHC on XPU: #32166
- [XPU] Use SYCL kernels for topk_transform on XPU: #33318
- [XPU] Use a fused GDN kernel from sgl-kernel for Qwen3.5 (Qwen3.5 GDN prefill 165 ms to 59 ms): #33354
- [Intel XPU] support prefill only models for xpu: #35072
- fix(xpu): enable compressed-tensors FP8 W8A8 on XPU (RedHatAI FP8-dynamic models): #33057
- XPU: remove SGLANG_USE_SGL_XPU flag: #34492
- [Intel XPU] Fix cross-encoder rerank hang on B580 runners: #36360
- [CPU] Enable ERNIE models on CPU: #35222
- [CPU] Fix NUMA/core binding for DP ranks: #32856
- Fix DP attention on CPU: #12961
- [CPU] Fix wrongly causal-masked bidirectional attention: #35434
- [CPU] Fix truncated KV prefix in intel_amx spec verify: #36430
- [CPU] Fix weight missing issue in fused_input_proj_cpu for GPTQ INT4 for Qwen 3.5: #35805
Security
- fix: bound CUDA memory for fast image preprocessing: #36295
- fix(frontend): bound stop strings and regex patterns: #37029
- [CI] Gate
/rerun-teston commenter trust and remove/rerun-stage: #35750
Dependencies
- Update deepep for SBO feature: #35450
- [Deps] Bump FlashInfer to 0.6.18: #36954 ⭐
- Use Flashinfer 0.6.18 release for CUDA 13.4 package: #37073
- Bump sgl-deep-gemm to v0.1.6: #36998
- Bump sgl-deep-gemm to 0.1.7: #37279
- chore: bump mooncake version to 0.3.13: #36493
- chore: bump tilelang to 0.1.12: #30874
- fix(deps): pin compressed-tensors to 0.18.0: #36794
- [NVIDIA] Add CUDA 13.4 container for initial Rubin support: #36233
Breaking Changes & Upgrade Notes
- The unified radix tree is now the default cache for every configuration, including Full-attention-only models;
SGLANG_ENABLE_UNIFIED_RADIX_TREEis deprecated and can be unset: #35081 - Spark3 is renamed Spark2.5 across config (
Spark2_5Config,model_typespark2_5), model classes, and the function-call parser; the tool-call parser name changes fromsparktospark25: #36416 - DeepSeek-V4 FP4 expert checkpoints default to the FlashInfer MXFP4 MoE runner on SM90, SM100, and SM120 when
--moe-runner-backendis left atauto; an explicit backend is preserved: #35919 - W4A4 MegaMoE moves to
--enable-w4a4-megamoe;SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTSandSGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KINDnow only warn: #35918 - FlashInfer 0.6.18 is required;
--dsa-topk-backend flashinferand the CuTe DSL NVFP4 W4A16 mode have no fallback for earlier FlashInfer APIs: #36954, #33237, #35120 - Constructing a
ServerArgsno longer resolves it. Programs that build their own record must callresolve_once()before reading resolved values; the launcher does this for the engine: #35907 - Kimi hybrid models honor
--enable-symm-memunder CUDA graphs again, since torch 2.13.0 fixes the allocator bug that forced it off: #34842 - Lean (work-centric) decode attention is auto-enabled on the ROCm Triton backend where its gate predicts a win, including under CUDA graphs; force it off with
--enable-lean-attention=FalseorSGLANG_DISABLE_LEAN_ATTENTION=1: #33576 - ROCm 7.2.4 images move to Python 3.12, torch 2.11, and triton 3.7; new ROCm 10 images ship for gfx942, gfx950, and gfx1250: #30984, #36434, #36871
- The
SGLANG_USE_SGL_XPUflag is removed; XPU now uses the sgl-kernel MoE kernels by default, and--moe-runner-backend tritonrestores the Triton path: #34492 - Diffusion Cache-DiT, CFG gating, and the DiT attention backend become per-request sampling params; the
SGLANG_CACHE_DIT_*env vars remain as server defaults and are slated for deprecation: #35339 - Hunyuan video VAE defaults to tiled decode: #36012
- Requests are limited to 32 stop strings and 32 stop regex patterns of at most 256 bytes each, returning HTTP 400 beyond that: #37029
/rerun-stageis removed and/rerun-testis gated on commenter trust (zero cooldown or repo write): #35750kill_process_treewaits for the reap by default in CLI, benchmark, and test teardowns: #36589
New Contributors
- @zetyquickly made their first contribution in #35107
- @vijay-kodamalla made their first contribution in #34327
- @HeYao0714 made their first contribution in #35162
- @jain-ria made their first contribution in #34627
- @yangweigbh made their first contribution in #35130
- @SubSir made their first contribution in #35371
- @QNan0513 made their first contribution in #34862
- @Dayuxiaoshui made their first contribution in #34993
- @MengYu10151 made their first contribution in #29525
- @ayush1399 made their first contribution in #32440
- @jason136 made their first contribution in #33606
- @ECMGit made their first contribution in #34679
- @shiyang814-cpu made their first contribution in #32327
- @wuyl1 made their first contribution in #33370
- @beyondHJM made their first contribution in #35361
- @wenkaidu made their first contribution in #32832
- @lgy1027 made their first contribution in #35796
- @daviswer made their first contribution in #35794
- @alecsolder made their first contribution in #35747
- @jianfei-wangg made their first contribution in #33829
- @fullyz made their first contribution in #35305
- @fatday made their first contribution in #34237
- @YangKai0616 made their first contribution in #30236
- @seokwoosong made their first contribution in #34066
- @cyxlily made their first contribution in #32166
- @ITerydh made their first contribution in #35829
- @unclezhou486 made their first contribution in #35676
- @jiaryang made their first contribution in #35672
- @cognaiger9 made their first contribution in #35188
- @shikicloud made their first contribution in #35505
- @yuanlehome made their first contribution in #35646
- @ActiveSky made their first contribution in #36275
- @FxxxxU made their first contribution in #36301
- @kuma-loong made their first contribution in #36465
- @Promisewjx made their first contribution in #33871
- @Deleter-D made their first contribution in #35275
- @galletas1712 made their first contribution in #34053
- @koppulac made their first contribution in #35290
- @kyx1999 made their first contribution in #36862
- @omirosh made their first contribution in #36852
- @tianxiaojiang4 made their first contribution in #36714
- @joeltg made their first contribution in #34639
- @tech-cow made their first contribution in #29100
- @zomglings made their first contribution in #37054
- @noron12234 made their first contribution in #33048
- @caihuali95 made their first contribution in #34602
- @ainergiz made their first contribution in #37035
- @TarangKhanna made their first contribution in #36101
- @Juhyun-Kim-Memphis made their first contribution in #37249
- @qtris123 made their first contribution in #33926
- @chien-an-chen made their first contribution in #34647
@ Every self-hosted release, weekly
One Monday email with the week's stable releases across the directory.
>_ Models sglang runs
Popular open-weight models with sglang support, with VRAM needs and licence at a glance.
More in LLM Inference
ShimmyA self-hosted, open-source LLM inference server: a single Rust binary that serves GGUF models through an OpenAI-compatible API, as an alternative to Ollama.
SmarterRouterSelf-hosted AI gateway that automatically routes queries to the best local or cloud LLM based on prompt analysis, with VRAM management and OpenAI-compatible API.
SNDR Core EngineSelf-hosted vLLM patch overlay running Qwen3.6 and Gemma 4 at 280K context on 24 GB GPUs with 1.5× throughput, working tool calls, and neural-graph memory.
tabbyAPITabbyAPI is a self-hosted, open-source FastAPI server providing an OpenAI-compatible REST API for running ExllamaV3-quantized LLMs on your own GPU hardware.
Transformer LabSelf-hosted, open-source ML platform to train, fine-tune, evaluate and serve LLMs, diffusion and audio models locally or across GPU clusters.
Turbo-LLMSelf-hosted local LLM platform running any engine with auto-tuning, real measured tokens/sec, and OpenAI/Anthropic APIs for Claude Code.
unslothSelf-hosted, open-source UI and framework to run, fine-tune and export LLMs, GGUF/MLX and diffusion models on your own GPU or Mac.
vLLMvLLM is a self-hosted, open-source LLM inference engine with state-of-the-art throughput via PagedAttention.
vllm-qwenSelf-hosted OpenAI-compatible inference server running Qwen3.6-27B BF16 on AMD Ryzen AI Max+ 395 with vision support, 256K context, and separated reasoning output.
WhisperLive---CPUSelf-hosted, open-source real-time Whisper transcription server with faster-whisper, TensorRT and OpenVINO backends and an OpenAI-compatible REST API.
YALSYALS is a self-hosted, open-source OpenAI-compatible API server for GGUF models via llama.cpp, with continuous batching and JSON schema support.
AGmindAGMind is a self-hosted, open-source installer deploying a private RAG stack on NVIDIA DGX Spark in ~30 minutes via an interactive wizard.

0 comments