5arXiv cs.CL (Computation and Language)·
EcoSpec: Cost-aware speculative decoding for MoE models reduces expert memory traffic
Researchers introduce EcoSpec, a speculative decoding framework that incorporates predicted marginal expert activation cost into draft-token selection for sparse Mixture-of-Experts LLMs. The key insight is that standard confidence-driven draft selection causes 'expert scattering'—routing draft tokens to disjoint experts increases memory traffic and undermines speculative decoding speedups. EcoSpec uses a lightweight expert predictor and dynamic expert buffer to favor draft paths that reuse already-loaded experts, achieving up to 1.62× end-to-end decoding speedup. Evaluations cover DeepSeek-V3.1 (671B), Qwen3-235B-A22B, and GPT-OSS-120B across reasoning, coding, QA, and dialogue tasks.
Related guides (3)
Related events (8)
DeLS-Spec: Decoupled long-short context speculative decoding improves drafting efficiency
DeLS-Spec is a new speculative decoding method that combines a fixed block-parallel draft model (DFlash) as a long-context expert with a lightweight locally-trained short-context head, avoiding joint training with the target model. The approach introduces intra-block causal conditioning at low training cost and is modular across DFlash checkpoints. Experiments on Qwen3 models show consistent speedup and acceptance-length improvements over DFlash on math, code, and dialogue benchmarks.
Expert Tying reduces MoE LLM memory footprint by ~2x with minimal quality loss
Researchers introduce Expert Tying, an architectural modification for Mixture-of-Experts LLMs that shares expert parameters across consecutive transformer layers while keeping routing and attention layer-independent. Evaluated on OLMoE, Qwen3, and DeepSeek-style MoE architectures, the method achieves nearly 2x memory reduction with negligible perplexity or downstream quality degradation. The approach exploits parameter redundancy in MoE pathways to improve the compute-to-memory trade-off for training and inference.
ZEDA: Post-Trained MoE Models Can Skip Half Their Experts via Self-Distillation
This paper introduces Zero-Expert Self-Distillation Adaptation (ZEDA), a framework that converts static post-trained Mixture-of-Experts (MoE) language models into dynamic ones without pre-training from scratch. ZEDA injects parameter-free zero-output experts into each MoE layer and uses two-stage self-distillation with the original model as a frozen teacher. Applied to Qwen3-30B-A3B and GLM-4.7-Flash across 11 benchmarks, ZEDA eliminates over 50% of expert FLOPs with marginal accuracy loss and achieves approximately 1.20× end-to-end inference speedup, outperforming the strongest dynamic MoE baseline by 4–6 points.
Practical investigation of training-free relaxed speculative decoding techniques
A new arXiv preprint systematically investigates training-free relaxed speculative decoding methods, which trade the strict distribution-preservation guarantee of standard speculative decoding for potential speed or capability gains. The authors unify existing approaches in a shared framework, benchmark them on contemporary settings, and surface practical findings for practitioners. Key takeaways include that relaxed approaches require careful capability evaluation and tend to depend on high-quality drafter models, making them poorly suited for lightweight multi-token-prediction drafters.
DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation for LLM Inference
DSpark is a new speculative decoding framework that combines a semi-autoregressive draft architecture with confidence-scheduled verification to improve LLM inference throughput. The semi-autoregressive design introduces intra-block token dependencies to reduce acceptance decay common in parallel drafters, while dynamic verification length tuning reduces wasted batch capacity in high-concurrency settings. Deployed within the DeepSeek-V4 serving system under live traffic, DSpark achieves 60–85% faster per-user generation speeds compared to the MTP-1 production baseline at matched throughput, and expands the Pareto frontier of latency-throughput tradeoffs.
DeepSeek open-sources DSpark speculative decoding module, achieving 50%+ throughput gains in production
Researchers from Peking University and DeepSeek introduced DSpark, a speculative decoding module that dynamically adjusts verification depth based on server load, achieving 57–85% faster per-user token generation and 51–52% higher total throughput compared to DeepSeek's previous production drafter. The team released checkpoints DeepSeek-V4-Pro-DSpark and DeepSeek-V4-Flash-DSpark on Hugging Face under an MIT license, with the draft module attaching to frozen target model weights. Key innovations include a parallel drafting backbone (adapted from DFlash), a Markov head for sequential token coherence correction, a calibrated confidence head, and a load-aware scheduler that trades draft length against server capacity. Results generalize across model families including Qwen3 and Gemma4.
VIA-SD: Multi-tier speculative decoding via intra-model routing cuts rejection rates and boosts inference speed
VIA-SD introduces a three-tier verification framework for speculative decoding that routes draft tokens to a lightweight 'slim verifier' submodel for medium-confidence cases, reserving full-model verification only for uncertain tokens. Across four tasks and multiple model families, the method reduces rejection rates by 0.10–0.22 and achieves 10–20% speedups over strong speculative decoding baselines, with 2.5–3x acceleration over standard decoding. The approach is compatible with existing speculative decoding frameworks without retraining. The work proposes multi-tier speculative decoding as a general paradigm for scalable LLM inference.
Mixture of Experts Explained
This Hugging Face blog post provides a technical overview of the Mixture of Experts (MoE) architecture, explaining how sparse gating mechanisms route tokens to subsets of expert feed-forward layers to achieve computational efficiency. The post covers training dynamics, inference considerations, and the tradeoffs between dense and sparse models. It serves as a reference document contextualizing MoE's growing relevance following high-profile model releases using the architecture.