6arXiv cs.CL (Computation and Language)·
Windowed-MTP: Eliminating Full-Context KV Cache Overhead in Speculative Decoding at Million-Token Scale
A new arXiv preprint introduces Windowed-MTP, a training-free modification to Multi-Token-Prediction draft heads used in speculative decoding that applies a StreamingLLM-style sliding window with attention sinks to the draft's attention only. At 1M-token context, the method drops ~99% of draft KV cache entries while leaving the full-attention verifier intact, making the approach lossless by construction. Evaluated on Qwen GDN-MoE 35B/122B and a Mamba2-hybrid 120B in SGLang, it reduces per-decode-step cost by 28–44% over the native MTP draft baseline. The fix addresses a compounding problem where deep native draft heads can become net-negative at long contexts, especially under hybrid/linear-attention targets.
Related guides (2)
Related events (8)
EcoSpec: Cost-aware speculative decoding for MoE models reduces expert memory traffic
Researchers introduce EcoSpec, a speculative decoding framework that incorporates predicted marginal expert activation cost into draft-token selection for sparse Mixture-of-Experts LLMs. The key insight is that standard confidence-driven draft selection causes 'expert scattering'—routing draft tokens to disjoint experts increases memory traffic and undermines speculative decoding speedups. EcoSpec uses a lightweight expert predictor and dynamic expert buffer to favor draft paths that reuse already-loaded experts, achieving up to 1.62× end-to-end decoding speedup. Evaluations cover DeepSeek-V3.1 (671B), Qwen3-235B-A22B, and GPT-OSS-120B across reasoning, coding, QA, and dialogue tasks.
DeLS-Spec: Decoupled long-short context speculative decoding improves drafting efficiency
DeLS-Spec is a new speculative decoding method that combines a fixed block-parallel draft model (DFlash) as a long-context expert with a lightweight locally-trained short-context head, avoiding joint training with the target model. The approach introduces intra-block causal conditioning at low training cost and is modular across DFlash checkpoints. Experiments on Qwen3 models show consistent speedup and acceptance-length improvements over DFlash on math, code, and dialogue benchmarks.
DepthWeave-KV: Token-adaptive cross-layer KV cache compression for long-context inference
A new arXiv preprint introduces DepthWeave-KV, a KV cache compression method that factorizes key-value states across neighboring transformer layers using shared low-rank channel bases while retaining token-specific residuals for attention-sensitive positions. A token-conditional depth router allocates higher reconstruction rank to instruction-bearing and retrieval-critical tokens, with calibration-free online error tracking during generation. The method achieves 8.3x KV memory reduction at 64K context while maintaining near-full-cache quality on LongBench, Needle-in-a-Haystack, and L-Eval benchmarks. The work addresses a practical bottleneck in long-context inference without requiring base model retraining.
Practical investigation of training-free relaxed speculative decoding techniques
A new arXiv preprint systematically investigates training-free relaxed speculative decoding methods, which trade the strict distribution-preservation guarantee of standard speculative decoding for potential speed or capability gains. The authors unify existing approaches in a shared framework, benchmark them on contemporary settings, and surface practical findings for practitioners. Key takeaways include that relaxed approaches require careful capability evaluation and tend to depend on high-quality drafter models, making them poorly suited for lightweight multi-token-prediction drafters.
VIA-SD: Multi-tier speculative decoding via intra-model routing cuts rejection rates and boosts inference speed
VIA-SD introduces a three-tier verification framework for speculative decoding that routes draft tokens to a lightweight 'slim verifier' submodel for medium-confidence cases, reserving full-model verification only for uncertain tokens. Across four tasks and multiple model families, the method reduces rejection rates by 0.10–0.22 and achieves 10–20% speedups over strong speculative decoding baselines, with 2.5–3x acceleration over standard decoding. The approach is compatible with existing speculative decoding frameworks without retraining. The work proposes multi-tier speculative decoding as a general paradigm for scalable LLM inference.
CLP: Lightweight collocation-length predictor achieves zero-loss multi-token inference speedup
Researchers propose CLP (Collocation-Length Predictor), a span-level decision layer for accelerating LLM inference via multi-token prediction without quality degradation. The key insight is 'Backbone-as-Architect': the backbone LM head always generates the first token while MTP heads handle only subsequent tokens, eliminating head-backbone competition that causes repetitive outputs in prior methods. CLP uses a single linear layer (~4.6K–7.7K parameters) versus 1M-parameter gate networks in prior work, achieving 1.14x–1.29x speedup on Qwen2.5 models with near-zero repetition ratio. The paper also establishes that shorter prediction horizons improve MTP head accuracy on larger models, offering a scaling-aware design principle.
AdaFlash: Adaptive speculative decoding with on-policy distilled diffusion drafters
AdaFlash is a new framework for accelerating LLM inference via speculative decoding that addresses variance problems in diffusion-based draft models. It combines an on-policy distillation algorithm using reverse-KL divergence with an adaptive length head that dynamically adjusts candidate sequence length. Experiments show up to ~66% higher throughput than prior state-of-the-art methods, with especially large gains in high-concurrency deployment scenarios.
SimSD: Speculative Decoding Adapted for Diffusion Language Models
SimSD introduces a training-free speculative decoding algorithm for diffusion large language models (dLLMs), which previously could not use standard token-level speculative decoding due to their bidirectional attention and masked language modeling formulation. The method uses a plug-and-play masking strategy that introduces reference tokens from a draft model and a custom attention mask, enabling valid logit computation for drafted tokens in a single forward pass. Evaluated on SDAR-family dLLMs across four benchmarks, SimSD achieves up to 7.46x decoding throughput improvement while maintaining or improving generation quality. The approach is compatible with other acceleration techniques such as KV cache and blockwise decoding.