EcoSpec

EcoSpec: Cost-aware speculative decoding for MoE models reduces expert memory traffic

Researchers introduce EcoSpec, a speculative decoding framework that incorporates predicted marginal expert activation cost into draft-token selection for sparse Mixture-of-Experts LLMs. The key insight is that standard confidence-driven draft selection causes 'expert scattering'—routing draft tokens to disjoint experts increases memory traffic and undermines speculative decoding speedups. EcoSpec uses a lightweight expert predictor and dynamic expert buffer to favor draft paths that reuse already-loaded experts, achieving up to 1.62× end-to-end decoding speedup. Evaluations cover DeepSeek-V3.1 (671B), Qwen3-235B-A22B, and GPT-OSS-120B across reasoning, coding, QA, and dialogue tasks.