4arXiv cs.AI (Artificial Intelligence)·
Practical investigation of training-free relaxed speculative decoding techniques
A new arXiv preprint systematically investigates training-free relaxed speculative decoding methods, which trade the strict distribution-preservation guarantee of standard speculative decoding for potential speed or capability gains. The authors unify existing approaches in a shared framework, benchmark them on contemporary settings, and surface practical findings for practitioners. Key takeaways include that relaxed approaches require careful capability evaluation and tend to depend on high-quality drafter models, making them poorly suited for lightweight multi-token-prediction drafters.
Related guides (2)
Related events (8)
Theory of acceptance criteria in speculative decoding for greedy and relaxed regimes
A new arXiv preprint develops a theoretical framework for speculative decoding acceptance criteria beyond the standard stochastic, distribution-preserving setting. The authors characterize rejection regions for greedy decoding, additive/multiplicative relaxed acceptance, top-m criteria, and entropy-thresholded acceptance in terms of KL divergence and margin-based bounds. The framework is extended to greedy tree decoding and validated empirically on Qwen3 models, showing relaxed and tree-based criteria substantially expand certified acceptance regions. The work fills a gap between existing theory and practical inference systems that use non-exact acceptance rules.
DeLS-Spec: Decoupled long-short context speculative decoding improves drafting efficiency
DeLS-Spec is a new speculative decoding method that combines a fixed block-parallel draft model (DFlash) as a long-context expert with a lightweight locally-trained short-context head, avoiding joint training with the target model. The approach introduces intra-block causal conditioning at low training cost and is modular across DFlash checkpoints. Experiments on Qwen3 models show consistent speedup and acceptance-length improvements over DFlash on math, code, and dialogue benchmarks.
VIA-SD: Multi-tier speculative decoding via intra-model routing cuts rejection rates and boosts inference speed
VIA-SD introduces a three-tier verification framework for speculative decoding that routes draft tokens to a lightweight 'slim verifier' submodel for medium-confidence cases, reserving full-model verification only for uncertain tokens. Across four tasks and multiple model families, the method reduces rejection rates by 0.10–0.22 and achieves 10–20% speedups over strong speculative decoding baselines, with 2.5–3x acceleration over standard decoding. The approach is compatible with existing speculative decoding frameworks without retraining. The work proposes multi-tier speculative decoding as a general paradigm for scalable LLM inference.
DominoTree: Training-free conditional tree drafting for speculative decoding achieves 6.6x LLM inference speedup
DominoTree is a new training-free speculative decoding method that constructs best-first draft trees scored by Domino's conditional, non-factorized GRU-based correction along each root-to-node path. On Qwen3-4B across eight benchmarks, it achieves up to 6.6x speedup over autoregressive decoding and a mean accept length of up to 10.7 tokens per round, outperforming prior methods including DDTree, CaDDTree, DFlash, and the base Domino decoder. A GPU-native CUDA-graph tree builder provides 9-10% throughput gains over Domino overall, with up to +22% on Alpaca, while maintaining bit-identical acceptance behavior.
SimSD: Speculative Decoding Adapted for Diffusion Language Models
SimSD introduces a training-free speculative decoding algorithm for diffusion large language models (dLLMs), which previously could not use standard token-level speculative decoding due to their bidirectional attention and masked language modeling formulation. The method uses a plug-and-play masking strategy that introduces reference tokens from a draft model and a custom attention mask, enabling valid logit computation for drafted tokens in a single forward pass. Evaluated on SDAR-family dLLMs across four benchmarks, SimSD achieves up to 7.46x decoding throughput improvement while maintaining or improving generation quality. The approach is compatible with other acceleration techniques such as KV cache and blockwise decoding.
Graft: Hybrid Tree Construction for Speculative Decoding via Prune-Then-Retrieve
Graft is a training-free framework that improves speculative decoding by coupling dynamic-depth pruning with retrieval-based token compensation. Pruning reduces VRAM and compute overhead while freeing budget for retrieval, which fills topological gaps in the draft tree with near-zero additional cost. On short-context benchmarks, Graft achieves up to 5.41× speedup and improves average speedup over EAGLE-3 by up to 21.8% on Qwen3-235B. The method is evaluated across short- and long-context settings and extended to block-drafting paradigms.
DeepSeek releases DSpark: speculative decoding system for LLM inference acceleration
DeepSeek published DSpark, a paper describing a speculative decoding system designed to accelerate LLM inference. The paper is hosted on DeepSeek's GitHub and attracted significant Hacker News engagement (598 points, 228 comments), suggesting meaningful community interest. Speculative decoding is an active inference optimization technique, and a release from DeepSeek carries weight given their track record on inference efficiency.
DeepSeek open-sources DSpark speculative decoding module, achieving 50%+ throughput gains in production
Researchers from Peking University and DeepSeek introduced DSpark, a speculative decoding module that dynamically adjusts verification depth based on server load, achieving 57–85% faster per-user token generation and 51–52% higher total throughput compared to DeepSeek's previous production drafter. The team released checkpoints DeepSeek-V4-Pro-DSpark and DeepSeek-V4-Flash-DSpark on Hugging Face under an MIT license, with the draft module attaching to frozen target model weights. Key innovations include a parallel drafting backbone (adapted from DFlash), a Markov head for sequential token coherence correction, a calibrated confidence head, and a load-aware scheduler that trades draft length against server capacity. Results generalize across model families including Qwen3 and Gemma4.