5arXiv cs.CL (Computation and Language)·
Randomized KV-cache eviction enables error certificates with provable coverage
A new arXiv paper proves that deterministic KV-cache eviction is fundamentally unidentifiable: evicted values can be manipulated so that retained state is unchanged while true attention-output error grows arbitrarily, making any serving-time error estimator inconsistent. The authors show that Poisson-sampled randomized eviction restores identifiability, enabling a Hájek-corrected survey-sampling variance estimator that serves as a per-step error certificate with 0.97 empirical coverage. Pre-registered experiments on real workloads show the certificate is most useful for attribution—separating cache-induced from inherent failures (AUC 0.73–0.75 vs. 0.47–0.54 for output confidence)—and for scheduling recomputation, though it does not improve predictive gating.
Related guides (2)
Related events (8)
VaSE: Value-Aware Stochastic KV Cache Eviction improves reasoning model efficiency
A new arXiv preprint introduces Value-aware Stochastic KV Cache Eviction (VaSE), a training-free method for compressing KV caches in long-chain-of-thought reasoning models. The authors identify two key failure modes in prior eviction approaches — catastrophic repetition loops caused by evicting high-magnitude value states, and low cache diversity — and address both with targeted protections and stochastic eviction. On six reasoning tasks with Qwen3 models at 4x compression, VaSE outperforms the current best selection-based sparse attention method and exceeds the strongest eviction baseline by over 4%, while supporting FlashAttention2 and maintaining a static memory footprint.
Fixed-lag smoothing framework for KV cache eviction: RMM policy shows gains only in specific reuse regimes
A new arXiv paper recasts KV cache eviction as a fixed-lag smoothing estimation problem, unifying existing methods (StreamingLLM, H2O, SnapKV, Belady's optimum) along a single 'commit lag' axis. The authors instantiate this framework as a training-free policy called RMM, a strict generalization of H2O that uses demonstrated token utility rather than accumulated attention. In controlled settings with endogenous reuse, RMM substantially outperforms baselines, but on standard third-party benchmarks inside NVIDIA's KVPress harness it performs on par with or below H2O and SnapKV. The paper's primary contribution is the theoretical framework and an honest characterization of when the measurement-based approach provides gains, explicitly disclaiming state-of-the-art status.
KV cache transplantation and precision controls explain stage-replay divergence in LLM inference
A preprint from arXiv investigates why stage-replay diagnostics for LLMs diverge from live inference, finding that KV cache state—not token identity—is the causally sufficient carrier of divergent trajectories in a Qwen2.5-derived system. A 200-item controlled experiment shows BF16 precision produces disagreements on 166 suffixes while FP32 eliminates decoded disagreements entirely. Bidirectional transplantation of all 48 KV layers causes continuations to follow the cache donor with 100% fidelity across two checkpoints (24/24 and 43/43). The findings have direct implications for reproducibility and debugging of multi-step reasoning systems.
Unlocking Longer Generation with Key-Value Cache Quantization
This Hugging Face blog post covers KV cache quantization as a technique to reduce memory consumption during LLM inference, enabling longer context generation without proportional VRAM increases. The post likely explains how quantizing the key-value cache (e.g., to INT8 or lower precision) trades minimal accuracy for significant memory savings. This is directly relevant to inference efficiency and long-context deployment patterns.
FreqDepthKV: Frequency-guided KV cache compression for long-context LLM inference
A new arXiv preprint introduces FreqDepthKV, an inference-time KV cache compression method that factorizes adjacent-layer KV states into shared low-frequency depth components and sparse high-frequency residuals. A lightweight online probe dynamically assigns attention heads to different caching modes without retraining. On a 32k-token prefill window, the method achieves a 3.9x compression ratio while closely matching full-KV accuracy across QA, retrieval, summarization, and code generation benchmarks, improving decoding throughput to 70.4 tokens/s and reducing peak KV memory to 6.2 GB.
DepthWeave-KV: Token-adaptive cross-layer KV cache compression for long-context inference
A new arXiv preprint introduces DepthWeave-KV, a KV cache compression method that factorizes key-value states across neighboring transformer layers using shared low-rank channel bases while retaining token-specific residuals for attention-sensitive positions. A token-conditional depth router allocates higher reconstruction rank to instruction-bearing and retrieval-critical tokens, with calibration-free online error tracking during generation. The method achieves 8.3x KV memory reduction at 64K context while maintaining near-full-cache quality on LongBench, Needle-in-a-Haystack, and L-Eval benchmarks. The work addresses a practical bottleneck in long-context inference without requiring base model retraining.
KVEraser: Learned KV cache editing for efficient localized context erasing in LLMs
KVEraser is a learned method for efficiently erasing specific spans from an LLM's KV cache without full recomputation of subsequent tokens. The approach replaces only the KV states of the erased interval with learned steering states, using a two-stage training pipeline of generic pre-training followed by task-specific fine-tuning. On contexts from 1K–32K tokens, KVEraser nearly matches full recomputation quality while incurring only 24% latency overhead versus a 17.6x increase for exact recomputation, with demonstrated generalization to long-document QA with harmful factual distractors.
Extrapolative Weight Averaging Reveals Correctness-Efficiency Frontiers in Code RL
This paper investigates whether extrapolative weight averaging of RL-trained checkpoints can extend Pareto frontiers between competing objectives (correctness vs. computational efficiency) without additional training. Starting from a shared initialization, the authors train checkpoints under nested unit-test coverage regimes for competitive programming tasks, revealing a correctness-efficiency frontier where higher-coverage rewards reduce optimization failures but increase correctness failures. Extrapolation beyond trained endpoints produces complementary policies that, when ensembled, improve pass@250 on LCB/hard by 3.3% over the best single checkpoint at matched sample budget. Results hold across 7B and 32B model scales and three inference settings: pure reasoning, tool use, and agentic coding.