technique

KV Cache Quantization

techniqueactivekv-cache-quantization-f8c8b092·1 events·first seen 28d ago

Aliases: KV Cache Quantization

Co-occurring entities

More like this (12)

KV Cache Vector Quantization scalar quantization quantization Channel-wise Vector Quantization 1.58-bit quantization binary quantization Value-aware Stochastic KV Cache Eviction INT4 Quantization quantization-induced degradation QVTo quantization-aware training

Recent events (1)

5Hugging Face Blog·28d ago·source ↗

Unlocking Longer Generation with Key-Value Cache Quantization

This Hugging Face blog post covers KV cache quantization as a technique to reduce memory consumption during LLM inference, enabling longer context generation without proportional VRAM increases. The post likely explains how quantizing the key-value cache (e.g., to INT8 or lower precision) trades minimal accuracy for significant memory savings. This is directly relevant to inference efficiency and long-context deployment patterns.

Training Infrastructure Long Context Evolution Transformers KV Cache Quantization Hugging Face +1 more