squad-07dc092e·2 events·first seen Aliases: SQuAD, SQuAD 2.0
A new arXiv preprint tests the implicit assumption that LLM evaluation is easier than generation, using a controlled in-context QA setup across four benchmarks (SQuAD 2.0, DROP, HotpotQA, MuSiQue) and two models. Results show generation accuracy exceeds self-evaluation accuracy on three of four benchmarks, with attention analysis revealing that evaluation attends to context 3–5x less than generation does. LoRA fine-tuning experiments confirm the asymmetry is not a training artifact, with cross-task interference observed in both directions. The findings directly challenge assumptions underlying LLM-as-a-Judge and self-evaluation pipelines widely used in RLHF and agentic systems.
This paper introduces a matched-control protocol to isolate whether RAG reader failures stem from context length or semantic competition among retrieved passages. By replacing hard-competitor passages with less competitive ones while holding passage count and length fixed, the authors demonstrate a measurable competition effect on SQuAD using Phi-2 and Qwen2.5-1.5B. Phi-2 recovers +6.0 EM and +7.0 answer-inclusion points; Qwen2.5-1.5B recovers +4.5 EM and +9.0 answer-inclusion points. The study also introduces retention curves and a right-censored half-life metric to track performance degradation as competitors accumulate.