correctness agreement

New paper exposes behavioral divergence in quantized LLMs beyond accuracy/perplexity metrics

A new arXiv preprint argues that standard accuracy and perplexity metrics fail to capture behavioral changes induced by post-training quantization. The authors introduce 'correctness agreement', a decision-level metric measuring overlap in correct predictions between base and quantized models, and find behavioral divergence emerges even when task performance appears preserved. Analysis of attention weight distortions reveals non-linear breakpoints at low bit-widths and that query/key projections are more sensitive to quantization than value/output projections. The findings challenge the assumption that quantized models are behaviorally equivalent to their base counterparts.