kvpress-6be260fc·2 events·first seen Aliases: KVPress
A new arXiv paper recasts KV cache eviction as a fixed-lag smoothing estimation problem, unifying existing methods (StreamingLLM, H2O, SnapKV, Belady's optimum) along a single 'commit lag' axis. The authors instantiate this framework as a training-free policy called RMM, a strict generalization of H2O that uses demonstrated token utility rather than accumulated attention. In controlled settings with endogenous reuse, RMM substantially outperforms baselines, but on standard third-party benchmarks inside NVIDIA's KVPress harness it performs on par with or below H2O and SnapKV. The paper's primary contribution is the theoretical framework and an honest characterization of when the measurement-based approach provides gains, explicitly disclaiming state-of-the-art status.
NVIDIA and Hugging Face present KVPress, a library for compressing the KV cache in large language models to enable more efficient long-context inference. The tool implements multiple KV cache compression ("pressing") algorithms that reduce memory footprint and latency without retraining models. KVPress is positioned as a practical toolkit for deploying LLMs in long-context scenarios where KV cache size becomes a bottleneck.