paper
Position Bias Correction is Insufficient for One-Pass Attention Sorting
paperactiveprovisional
position-bias-correction-is-insufficient-for-one-pass-attention-sorting-ca84eb4d·1 events·first seen 41h agoAliases: Position Bias Correction is Insufficient for One-Pass Attention Sorting
Co-occurring entities
More like this (12)
Debiased One-Pass Attention SortingAttention SortingDifferential AttentionProbSparse AttentionBias Benchmark for Question AnsweringLie-Algebra Attentionpositional attention headsSliding Window Attentionbidirectional attentionUnBias-PlusBlock Sparse AttentionAttention Amnesia in Hybrid LLMs: When CoT Fine-Tuning Breaks Long-Range Recall, and How to Fix It
Recent events (1)
Debiased One-Pass Attention Sorting fails to close gap with iterative sorting for long-context LLMs
A new arXiv preprint investigates whether position bias is the primary bottleneck in long-context LLM performance, proposing Debiased One-Pass Attention Sorting as a cheaper alternative to iterative Attention Sorting. Experiments on LLaMA-2-7B-32K-Instruct and YaRN-Llama-2-7b-64k show that bias correction alone is insufficient: on one model it provides no improvement over uncalibrated single-pass sorting, and on the other it closes only 37% of the gap to iterative sorting. The findings suggest that iterative reordering provides benefits beyond position-bias correction, leaving the efficiency-accuracy tradeoff unresolved.