acl-anthology-7c0ed763·2 events·first seen Aliases: ACL Anthology
A new arXiv paper surveys 650 ACL Anthology papers that use LLM-as-a-Judge evaluation, finding only 33 address multilingual or low-resource language settings. Analysis of those 33 papers reveals inconsistent outcomes, overtrust in LLM judgments, and over-reliance on single judge models. The authors provide recommendations for improving evaluation practice in these underserved settings.
The paper introduces ACL-Verbatim, an extractive question answering system built on VerbatimRAG that maps user queries directly to verbatim text spans in ACL Anthology papers, eliminating hallucination by design. The authors contribute a new ground-truth benchmark dataset created via human NLP-researcher annotation over synthetic queries generated using a ScIRGen-based pipeline. A 150M-parameter ModernBERT token classifier trained on silver supervision achieves the best word-level F1 of 53.6, outperforming the strongest LLM-based extractor at 48.7. The work demonstrates that smaller extractive models can outperform large generative LLMs on precision-critical retrieval tasks.