freqdepthkv-f2b89145·1 events·first seen Aliases: FreqDepthKV
A new arXiv preprint introduces FreqDepthKV, an inference-time KV cache compression method that factorizes adjacent-layer KV states into shared low-frequency depth components and sparse high-frequency residuals. A lightweight online probe dynamically assigns attention heads to different caching modes without retraining. On a 32k-token prefill window, the method achieves a 3.9x compression ratio while closely matching full-KV accuracy across QA, retrieval, summarization, and code generation benchmarks, improving decoding throughput to 70.4 tokens/s and reducing peak KV memory to 6.2 GB.