Entity · paper

When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability

paperactiveprovisionalwhen-the-judge-changes-so-does-the-measurement-auditing-llm-as-judge-reliability-2b395b7c·1 events·first seen 3h ago

Aliases: When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability

Co-occurring entities

MiniMax Alibaba Qwen3

More like this (12)

LLM-judge scoring Can LLMs Judge Better Than They Generate? Evaluating Task Asymmetry, Mechanistic Interpretability and Transferability for In-Context QA LLM-as-a-Judge Challenges and Recommendations for LLMs-as-a-Judge in Multilingual Settings and Low-Resource Languages LLM-judged explanation score Beyond Third-Person Audits: Situated Interaction Auditing for User-Centered LLM Bias Research Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance?Judgement-of-Learning Clinician-Level Agreement Without Clinical Caution: LLM Evaluator Limits in Medical AI Benchmarking Surrogate Fidelity: When Can Open LLMs Explain Closed Ones?JAM (Judge for Adaptive Metric-Alignment)LLM-as-a-Verifier

Recent events (1)

6arXiv · cs.CL·3h ago·source ↗

Auditing LLM-as-Judge reliability: judge upgrades are not interchangeable across model families

A new arXiv paper investigates measurement validity problems in LLM-as-judge evaluation, finding that swapping evaluator models changes scores even when candidate responses are fixed. Across four judgment datasets, the authors compare Qwen3 dense judges (1.7B–32B) and MiniMax M2/M2.7 API releases, finding that only the Qwen3 1.7B→4B upgrade yields robust adjacent gains while MiniMax adjacent releases do not. Stronger judges reduce but do not eliminate position and verbosity bias, and repeated-sample juries add little when errors are correlated. The paper argues for standardized reporting requirements including dataset slices, bias probes, error-dependence estimates, and protocol audit trails.

Evaluation and Benchmarking AI Safety Research When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability MiniMax Alibaba +1 more