snapkv-7f299980·2 events·first seen Aliases: SnapKV
A new arXiv paper recasts KV cache eviction as a fixed-lag smoothing estimation problem, unifying existing methods (StreamingLLM, H2O, SnapKV, Belady's optimum) along a single 'commit lag' axis. The authors instantiate this framework as a training-free policy called RMM, a strict generalization of H2O that uses demonstrated token utility rather than accumulated attention. In controlled settings with endogenous reuse, RMM substantially outperforms baselines, but on standard third-party benchmarks inside NVIDIA's KVPress harness it performs on par with or below H2O and SnapKV. The paper's primary contribution is the theoretical framework and an honest characterization of when the measurement-based approach provides gains, explicitly disclaiming state-of-the-art status.
ReasonAlloc is a training-free framework that reframes decoding-time KV cache compression as a hierarchical budget allocation problem, operating at both layer-wise (offline) and head-wise (online) levels. The method identifies an architecture-driven pattern called the 'Reasoning Wave' to guide layer preallocation, then dynamically reallocates to information-rich heads during decoding. Evaluated on MATH-500 and AIME 2024 using DeepSeek-R1-Distill and AceReason models, it outperforms uniform-budget baselines (R-KV, SnapKV, Pyramid-RKV) especially at small budgets of 128–512 tokens, with negligible overhead.