6arXiv cs.CL (Computation and Language)·
LLM judges over-credit incorrect answers in no-reference evaluation settings
A new arXiv paper investigates the reliability of LLM-as-judge evaluation in no-reference settings, finding that judge models systematically over-credit incorrect answers when no ground-truth is provided. Sensitivity experiments across three languages show that adding reference answers to prompts flips correct/incorrect decisions by up to 85% in some settings, with these changes aligning with human annotations. The authors propose a calibration methodology—testing judge knowledge with reference-aware samples before deploying in reference-free setups—as a blueprint for practitioners.
Related guides (2)
Related events (8)
Survey and recommendations for LLM-as-a-Judge evaluation in multilingual and low-resource language settings
A new arXiv paper surveys 650 ACL Anthology papers that use LLM-as-a-Judge evaluation, finding only 33 address multilingual or low-resource language settings. Analysis of those 33 papers reveals inconsistent outcomes, overtrust in LLM judgments, and over-reliance on single judge models. The authors provide recommendations for improving evaluation practice in these underserved settings.
Auditing LLM-as-Judge reliability: judge upgrades are not interchangeable across model families
A new arXiv paper investigates measurement validity problems in LLM-as-judge evaluation, finding that swapping evaluator models changes scores even when candidate responses are fixed. Across four judgment datasets, the authors compare Qwen3 dense judges (1.7B–32B) and MiniMax M2/M2.7 API releases, finding that only the Qwen3 1.7B→4B upgrade yields robust adjacent gains while MiniMax adjacent releases do not. Stronger judges reduce but do not eliminate position and verbosity bias, and repeated-sample juries add little when errors are correlated. The paper argues for standardized reporting requirements including dataset slices, bias probes, error-dependence estimates, and protocol audit trails.
Benchmarking LLM judges for citation verification in deep-research systems
A new arXiv paper evaluates 8 LLM judges from 3 model families on citation quality assessment for deep-research systems, testing across 1,248 rubric decisions with human-reviewed gold labels. The study finds that cheaper models remain competitive with frontier models — GPT-5-mini achieves the strongest source-relevance F1 at 0.908 — but judges differ substantially in directional bias (pass-rate drift, false positive/negative rates) even when scalar F1 scores are similar. The key finding is that scalar F1 obscures biases that would be directly reinforced in an RL training loop, making judge calibration a prerequisite before using citation rubrics as reward signals.
LLMs judge worse than they generate: empirical challenge to self-evaluation pipeline assumptions
A new arXiv preprint tests the implicit assumption that LLM evaluation is easier than generation, using a controlled in-context QA setup across four benchmarks (SQuAD 2.0, DROP, HotpotQA, MuSiQue) and two models. Results show generation accuracy exceeds self-evaluation accuracy on three of four benchmarks, with attention analysis revealing that evaluation attends to context 3–5x less than generation does. LoRA fine-tuning experiments confirm the asymmetry is not a training artifact, with cross-task interference observed in both directions. The findings directly challenge assumptions underlying LLM-as-a-Judge and self-evaluation pipelines widely used in RLHF and agentic systems.
Towards Reliable Multilingual LLMs-as-a-Judge: An Empirical Study
This paper systematically investigates strategies for extending LLM-based automatic evaluation (LLMs-as-a-Judge) to multilingual settings, covering high-, mid-, and low-resource languages (English, Spanish, Basque). The authors compare instruction translation, monolingual vs. multilingual supervision, and model size, finding that fine-tuned smaller models can match proprietary models when in-domain data is available, while zero-shot larger models are preferable out-of-domain. Two meta-evaluation datasets are extended to Spanish and Basque, and all data and code are publicly released.
Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modeling
This paper identifies and analyzes 'Perceptual Judgment Bias' in multimodal LLM judges, where models anchor on response text rather than visual evidence when the two conflict. The authors introduce a Perceptually Perturbed Judgment Dataset using counterfactual responses to isolate perceptual errors, and a training framework combining GRPO-based reward modeling with batch-ranking objectives. Experiments on MLLM-as-a-Judge benchmarks show improved perceptual fidelity, ranking coherence, and alignment with human evaluation.
Systematic study of LLM data referencing errors in tables, with lightweight critic model mitigation
A new arXiv paper introduces the first systematic evaluation of data referencing errors (DREs) — incorrect citation or omission of table values — across LLMs ranging from 1.7B to 20B parameters. The authors find DREs are pervasive across all tested models and tasks, compromising intermediate reasoning steps beyond just final-answer accuracy. They demonstrate that a critic-based filtering and rejection sampling approach improves answer accuracy by up to 12%, and train a lightweight 4B critic model achieving 78.2% F1 on detecting DREs both in- and out-of-distribution.
Paper challenges LLM expert-level claims by measuring variance and error magnitude in code-based data analysis tasks
A new arXiv paper argues that standard LLM benchmarks overstate model capabilities by focusing on average performance on training-data-adjacent tasks while ignoring response variance and error magnitude. The authors introduce a novel benchmark requiring frontier LLMs to write code for data analysis tasks, comparing results against human expert submissions. Human experts outperformed the frontier LLM on average across multiple metrics and showed lower performance variability. The findings challenge the prevailing narrative that LLMs perform at human-expert level on knowledge economy tasks.