Entity · paper

LLM Judges Can Be Too Generous When There Is No Reference Answer

paperactiveprovisionalllm-judges-can-be-too-generous-when-there-is-no-reference-answer-0979d54a·1 events·first seen 14h ago

Aliases: LLM Judges Can Be Too Generous When There Is No Reference Answer

More like this (12)

Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias LLM-judged explanation score LLM-as-a-Judge When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability LLM-judge scoring Challenges and Recommendations for LLMs-as-a-Judge in Multilingual Settings and Low-Resource Languages Can LLMs Judge Better Than They Generate? Evaluating Task Asymmetry, Mechanistic Interpretability and Transferability for In-Context QA When LLMs Read Tables Carelessly: Measuring and Reducing Data Referencing Errors Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance?Surrogate Fidelity: When Can Open LLMs Explain Closed Ones?Two Axes of LLM Abstention: Answer Correctness and Question Answerability Can LLMs Reliably Self-Report Adversarial Prefills, and How?

Recent events (1)

6arXiv · cs.CL·14h ago·source ↗

LLM judges over-credit incorrect answers in no-reference evaluation settings

A new arXiv paper investigates the reliability of LLM-as-judge evaluation in no-reference settings, finding that judge models systematically over-credit incorrect answers when no ground-truth is provided. Sensitivity experiments across three languages show that adding reference answers to prompts flips correct/incorrect decisions by up to 85% in some settings, with these changes aligning with human annotations. The authors propose a calibration methodology—testing judge knowledge with reference-aware samples before deploying in reference-free setups—as a blueprint for practitioners.

Evaluation and Benchmarking AI Safety Research LLM Judges Can Be Too Generous When There Is No Reference Answer