Entity · technique

StreamingLLM

techniqueactiveprovisionalstreamingllm-82661e2b·1 events·first seen 3d ago

Aliases: StreamingLLM

Co-occurring entities

Windowed-MTP SGLang Mamba2-hybrid NoPE 120B Qwen GDN-MoE 35B

More like this (12)

SpeechLLM EvalLLM Audio-LLM RTLLM Dep-LLM LLM Wiki whichllm vLLM SVD-LLM LLM frontier LLMs LLM CLI

Recent events (1)

6arXiv · cs.CL·3d ago·source ↗

Windowed-MTP: Eliminating Full-Context KV Cache Overhead in Speculative Decoding at Million-Token Scale

A new arXiv preprint introduces Windowed-MTP, a training-free modification to Multi-Token-Prediction draft heads used in speculative decoding that applies a StreamingLLM-style sliding window with attention sinks to the draft's attention only. At 1M-token context, the method drops ~99% of draft KV cache entries while leaving the full-attention verifier intact, making the approach lossless by construction. Evaluated on Qwen GDN-MoE 35B/122B and a Mamba2-hybrid 120B in SGLang, it reduces per-decode-step cost by 28–44% over the native MTP draft baseline. The fix addresses a compounding problem where deep native draft heads can become net-negative at long contexts, especially under hybrid/linear-attention targets.

Long Context Evolution Inference Economics Windowed-MTP SGLang StreamingLLM +2 more