5arXiv cs.CL (Computation and Language)·
KV cache transplantation and precision controls explain stage-replay divergence in LLM inference
A preprint from arXiv investigates why stage-replay diagnostics for LLMs diverge from live inference, finding that KV cache state—not token identity—is the causally sufficient carrier of divergent trajectories in a Qwen2.5-derived system. A 200-item controlled experiment shows BF16 precision produces disagreements on 166 suffixes while FP32 eliminates decoded disagreements entirely. Bidirectional transplantation of all 48 KV layers causes continuations to follow the cache donor with 100% fidelity across two checkpoints (24/24 and 43/43). The findings have direct implications for reproducibility and debugging of multi-step reasoning systems.
Related guides (2)
Related events (8)
DepthWeave-KV: Token-adaptive cross-layer KV cache compression for long-context inference
A new arXiv preprint introduces DepthWeave-KV, a KV cache compression method that factorizes key-value states across neighboring transformer layers using shared low-rank channel bases while retaining token-specific residuals for attention-sensitive positions. A token-conditional depth router allocates higher reconstruction rank to instruction-bearing and retrieval-critical tokens, with calibration-free online error tracking during generation. The method achieves 8.3x KV memory reduction at 64K context while maintaining near-full-cache quality on LongBench, Needle-in-a-Haystack, and L-Eval benchmarks. The work addresses a practical bottleneck in long-context inference without requiring base model retraining.
FreqDepthKV: Frequency-guided KV cache compression for long-context LLM inference
A new arXiv preprint introduces FreqDepthKV, an inference-time KV cache compression method that factorizes adjacent-layer KV states into shared low-frequency depth components and sparse high-frequency residuals. A lightweight online probe dynamically assigns attention heads to different caching modes without retraining. On a 32k-token prefill window, the method achieves a 3.9x compression ratio while closely matching full-KV accuracy across QA, retrieval, summarization, and code generation benchmarks, improving decoding throughput to 70.4 tokens/s and reducing peak KV memory to 6.2 GB.
Unlocking Longer Generation with Key-Value Cache Quantization
This Hugging Face blog post covers KV cache quantization as a technique to reduce memory consumption during LLM inference, enabling longer context generation without proportional VRAM increases. The post likely explains how quantizing the key-value cache (e.g., to INT8 or lower precision) trades minimal accuracy for significant memory savings. This is directly relevant to inference efficiency and long-context deployment patterns.
KVEraser: Learned KV cache editing for efficient localized context erasing in LLMs
KVEraser is a learned method for efficiently erasing specific spans from an LLM's KV cache without full recomputation of subsequent tokens. The approach replaces only the KV states of the erased interval with learned steering states, using a two-stage training pipeline of generic pre-training followed by task-specific fine-tuning. On contexts from 1K–32K tokens, KVEraser nearly matches full recomputation quality while incurring only 24% latency overhead versus a 17.6x increase for exact recomputation, with demonstrated generalization to long-document QA with harmful factual distractors.
ReContext: Training-free recursive evidence replay improves LLM long-context reasoning
Researchers introduce RECONTEXT, a training-free inference-time method for improving long-context reasoning in LLMs. The approach uses model-internal relevance signals to build a query-conditioned evidence pool that is replayed before final generation, without modifying the original context, external memory, or context pruning. Experiments across eight long-context datasets at 128K context length show consistent improvements on Qwen3-4B, Qwen3-8B, and Llama3-8B. The authors provide a theoretical grounding via associative memory theory, framing attention as cue-trace association and replay as trace reactivation.
VaSE: Value-Aware Stochastic KV Cache Eviction improves reasoning model efficiency
A new arXiv preprint introduces Value-aware Stochastic KV Cache Eviction (VaSE), a training-free method for compressing KV caches in long-chain-of-thought reasoning models. The authors identify two key failure modes in prior eviction approaches — catastrophic repetition loops caused by evicting high-magnitude value states, and low cache diversity — and address both with targeted protections and stochastic eviction. On six reasoning tasks with Qwen3 models at 4x compression, VaSE outperforms the current best selection-based sparse attention method and exceeds the strongest eviction baseline by over 4%, while supporting FlashAttention2 and maintaining a static memory footprint.
LCGuard: Adversarial Training Framework for Safe KV Cache Sharing in Multi-Agent LLM Systems
LCGuard introduces a framework for preventing sensitive information leakage when multi-agent LLM systems share KV caches as a latent communication channel. The approach formalizes leakage operationally via reconstruction: a shared cache artifact is deemed unsafe if an adversarial decoder can recover sensitive inputs from it. An adversarial training loop pits a reconstructor against LCGuard's representation-level transformations, which aim to preserve task-relevant semantics while suppressing recoverable sensitive content. Empirical results across multiple model families and multi-agent benchmarks show reduced reconstruction-based leakage and attack success rates with competitive task performance.
QK-Restore: Fixing long-context recall degradation caused by CoT fine-tuning in hybrid LLMs
Researchers find that chain-of-thought supervised fine-tuning systematically degrades long-context recall in hybrid linear-attention models (HypeNet, Jet-Nemotron), with Needle-In-A-Haystack performance collapsing dramatically—e.g., HypeNet-9B dropping from 67.2% to 9.4% at 256K context. The root cause is identified as CoT-SFT biasing attention gradients toward short-range patterns, corrupting the query-key projections responsible for long-range routing. The paper proposes QK-Restore, a training-free fix that restores only W_Q and W_K from the pre-SFT checkpoint, recovering long-context capability while preserving reasoning gains.