5arXiv cs.AI (Artificial Intelligence)·
Fixed-lag smoothing framework for KV cache eviction: RMM policy shows gains only in specific reuse regimes
A new arXiv paper recasts KV cache eviction as a fixed-lag smoothing estimation problem, unifying existing methods (StreamingLLM, H2O, SnapKV, Belady's optimum) along a single 'commit lag' axis. The authors instantiate this framework as a training-free policy called RMM, a strict generalization of H2O that uses demonstrated token utility rather than accumulated attention. In controlled settings with endogenous reuse, RMM substantially outperforms baselines, but on standard third-party benchmarks inside NVIDIA's KVPress harness it performs on par with or below H2O and SnapKV. The paper's primary contribution is the theoretical framework and an honest characterization of when the measurement-based approach provides gains, explicitly disclaiming state-of-the-art status.
Related guides (3)
Related events (8)
Randomized KV-cache eviction enables error certificates with provable coverage
A new arXiv paper proves that deterministic KV-cache eviction is fundamentally unidentifiable: evicted values can be manipulated so that retained state is unchanged while true attention-output error grows arbitrarily, making any serving-time error estimator inconsistent. The authors show that Poisson-sampled randomized eviction restores identifiability, enabling a Hájek-corrected survey-sampling variance estimator that serves as a per-step error certificate with 0.97 empirical coverage. Pre-registered experiments on real workloads show the certificate is most useful for attribution—separating cache-induced from inherent failures (AUC 0.73–0.75 vs. 0.47–0.54 for output confidence)—and for scheduling recomputation, though it does not improve predictive gating.
VaSE: Value-Aware Stochastic KV Cache Eviction improves reasoning model efficiency
A new arXiv preprint introduces Value-aware Stochastic KV Cache Eviction (VaSE), a training-free method for compressing KV caches in long-chain-of-thought reasoning models. The authors identify two key failure modes in prior eviction approaches — catastrophic repetition loops caused by evicting high-magnitude value states, and low cache diversity — and address both with targeted protections and stochastic eviction. On six reasoning tasks with Qwen3 models at 4x compression, VaSE outperforms the current best selection-based sparse attention method and exceeds the strongest eviction baseline by over 4%, while supporting FlashAttention2 and maintaining a static memory footprint.
KV cache transplantation and precision controls explain stage-replay divergence in LLM inference
A preprint from arXiv investigates why stage-replay diagnostics for LLMs diverge from live inference, finding that KV cache state—not token identity—is the causally sufficient carrier of divergent trajectories in a Qwen2.5-derived system. A 200-item controlled experiment shows BF16 precision produces disagreements on 166 suffixes while FP32 eliminates decoded disagreements entirely. Bidirectional transplantation of all 48 KV layers causes continuations to follow the cache donor with 100% fidelity across two checkpoints (24/24 and 43/43). The findings have direct implications for reproducibility and debugging of multi-step reasoning systems.
VideoMLA: Low-Rank Latent KV Cache for Minute-Scale Autoregressive Video Diffusion
VideoMLA applies Multi-Head Latent Attention (MLA) to causal video diffusion, replacing per-head keys and values with a shared low-rank content latent and decoupled 3D-RoPE positional key, achieving 92.7% reduction in per-token KV memory. The paper investigates why MLA works despite pretrained video attention not being low-rank (unlike the spectral assumption motivating MLA in LLMs), finding that the MLA bottleneck itself determines effective rank rather than the pretrained spectrum. On VBench, VideoMLA matches short-horizon baselines, achieves best overall score at long horizons, and delivers 1.23x throughput improvement on a single NVIDIA B200 GPU.
KVEraser: Learned KV cache editing for efficient localized context erasing in LLMs
KVEraser is a learned method for efficiently erasing specific spans from an LLM's KV cache without full recomputation of subsequent tokens. The approach replaces only the KV states of the erased interval with learned steering states, using a two-stage training pipeline of generic pre-training followed by task-specific fine-tuning. On contexts from 1K–32K tokens, KVEraser nearly matches full recomputation quality while incurring only 24% latency overhead versus a 17.6x increase for exact recomputation, with demonstrated generalization to long-document QA with harmful factual distractors.
JoLT: Near-lossless KV cache compression via joint Tucker decomposition and JL-residual allocation
Researchers introduce JoLT, a KV cache compression method that treats the cache as a third-order tensor and applies a partial Tucker decomposition on the token and feature axes, then recovers truncation error with a Johnson-Lindenstrauss rotated low-bit residual. A Lagrangian dual jointly allocates Tucker ranks and residual bit-widths per layer group under a single byte budget. The method achieves 2-3x near-lossless compression on Mistral-7B-v0.3 and LLaMA-2-13B, with Frobenius reconstruction error roughly an order of magnitude below cross-layer SVD and 4-bit quantization. A randomized-SVD variant, FlashJoLT, delivers 5-13x compression-time speedup at matched quality.
FreqDepthKV: Frequency-guided KV cache compression for long-context LLM inference
A new arXiv preprint introduces FreqDepthKV, an inference-time KV cache compression method that factorizes adjacent-layer KV states into shared low-frequency depth components and sparse high-frequency residuals. A lightweight online probe dynamically assigns attention heads to different caching modes without retraining. On a 32k-token prefill window, the method achieves a 3.9x compression ratio while closely matching full-KV accuracy across QA, retrieval, summarization, and code generation benchmarks, improving decoding throughput to 70.4 tokens/s and reducing peak KV memory to 6.2 GB.
Unlocking Longer Generation with Key-Value Cache Quantization
This Hugging Face blog post covers KV cache quantization as a technique to reduce memory consumption during LLM inference, enabling longer context generation without proportional VRAM increases. The post likely explains how quantizing the key-value cache (e.g., to INT8 or lower precision) trades minimal accuracy for significant memory savings. This is directly relevant to inference efficiency and long-context deployment patterns.