6arXiv cs.CL (Computation and Language)·
Structural test oracle problem in LLM-as-judge synthetic corpora: silent failures and a validation protocol
A new arXiv paper reports a case where a shared decoding-budget parameter silently truncated hallucinated answers in a multilingual LLM-as-judge evaluation corpus, producing a spurious 32-point cross-lingual accuracy collapse that replicated robustly across sample sizes but was entirely artifactual. The authors argue this failure mode is structural to LLM-generated negative examples, which lack any mechanical item-level integrity check (the 'test oracle problem'), unlike corpora built from deterministic perturbation of gold answers. A second real bias (Markdown formatting preference) was simultaneously distorted in magnitude and sign by the same fault, illustrating that aggregate statistics cannot distinguish fabricated from distorted effects. The paper closes with a validation protocol for analysts working with oracle-less multilingual LLM-as-judge corpora.
Related guides (2)
Related events (8)
BabelJudge: Benchmark for measuring LLM-as-a-judge reliability across languages and agent trajectories
BabelJudge is a new open-source benchmark and audit framework that systematically measures four failure modes in LLM-as-a-judge systems: position bias, verbosity bias, order inconsistency, and cross-lingual degradation. The framework uses a 'gold-labelling by degradation' technique to generate labeled evaluation pairs without human annotation. Evaluation of Qwen2.5-7B-Instruct-4bit across English, Hindi, Arabic, and Swahili reveals severe cross-lingual reliability drops, with Swahili order consistency collapsing to near-random (0.480). The framework is extended to agentic evaluation with nine trajectory-level perturbations and three new metrics, released as a Python package supporting 11 judge backends.
LLM judges over-credit incorrect answers in no-reference evaluation settings
A new arXiv paper investigates the reliability of LLM-as-judge evaluation in no-reference settings, finding that judge models systematically over-credit incorrect answers when no ground-truth is provided. Sensitivity experiments across three languages show that adding reference answers to prompts flips correct/incorrect decisions by up to 85% in some settings, with these changes aligning with human annotations. The authors propose a calibration methodology—testing judge knowledge with reference-aware samples before deploying in reference-free setups—as a blueprint for practitioners.
Audit reveals protocol-level shortcuts in large audio-language model judges for speech evaluation
Researchers audit 'protocol-level shortcuts' in large audio-language models (LALMs) used as automatic judges for speech evaluation, testing across three deployment protocols: feature-blueprint judging, reference-conditioned judging, and pairwise A/B comparison. Across six judges and four attributes, several LALMs are found to rely on shortcuts rather than actual audio content — for example, incorrect specialist labels collapse emotion accuracy to 0.10 or below for five judges, and Qwen3-Omni-Thinking shows position bias in A/B comparisons. The findings indicate that high human-agreement scores can overstate judge validity, and that each model-protocol pair requires its own shortcut probe to be trustworthy.
Auditing LLM-as-Judge reliability: judge upgrades are not interchangeable across model families
A new arXiv paper investigates measurement validity problems in LLM-as-judge evaluation, finding that swapping evaluator models changes scores even when candidate responses are fixed. Across four judgment datasets, the authors compare Qwen3 dense judges (1.7B–32B) and MiniMax M2/M2.7 API releases, finding that only the Qwen3 1.7B→4B upgrade yields robust adjacent gains while MiniMax adjacent releases do not. Stronger judges reduce but do not eliminate position and verbosity bias, and repeated-sample juries add little when errors are correlated. The paper argues for standardized reporting requirements including dataset slices, bias probes, error-dependence estimates, and protocol audit trails.
Aggregate accuracy masks per-example prediction instability from task-irrelevant context in LLMs
A new arXiv paper demonstrates that state-of-the-art LLMs appear robust to task-irrelevant context at the aggregate level, but this stability conceals significant per-example prediction flips. Even semantically meaningless pseudo-words prepended to benchmark questions can shift model predictions on a subset of examples, with gains and losses roughly canceling out in aggregate. The instability is modulated by context type, length, test-time compute, and model development stage, and the affected examples are largely model-specific. The authors argue this reveals 'tail risks' hidden by standard aggregate accuracy metrics, motivating per-example reliability evaluation.
AMALIA-9B study finds LLM annotation agreement does not imply construct validity for moral foundation coding
A new arXiv paper tests whether Portugal's publicly funded AMALIA-9B language model can validly annotate moral foundation constructs (specifically authority) in European Portuguese text, not just agree with human coders. Using a 'recovery gap' methodology — comparing holistic prompting against decomposed codebook-clause prompting — the authors find AMALIA recovers only about half its holistic performance under decomposition, suggesting reliance on surface correlates rather than genuine construct understanding. A larger open multilingual model closes the same gap on the same corpus, pointing to model-level rather than corpus-level failure. The paper argues that sovereign/national LLM benchmark batteries should test the evidential route to agreement, not just agreement rates.
Semantic vs. Surface Noise in LLM Agents: 68-Cell Measurement Study with Held-Out Validation
This paper documents an empirical phenomenon across 10 LLMs from 7 architecture families: meaning-bearing perturbations (paraphrase, synonym substitution) cause final-answer inconsistency ~19.69 percentage points more often than presentation-level perturbations (formatting, reordering) of comparable severity, across GSM8K, MATH, and HotpotQA benchmarks. The effect is validated on a held-out 11th model (qwen2.5-14B-Instruct) with 1,800 trajectories. Trace-level analysis supports a 'stealth-divergence' picture where semantic perturbations preserve the first action but induce divergence in intermediate reasoning steps, while two prior mechanism claims are explicitly retracted. The study is notable for its honest reporting of stress-test failures and pre-registered replication.
Mechanistic interpretability study reveals geometric structure of LLM-as-judge bias in hidden states
A new arXiv preprint investigates LLM-as-judge scoring bias at the representation level rather than the input-output level, studying seven judge models across seven bias types and nine benchmarks. The authors find that biased inputs are displaced along low-dimensional, type-specific subspaces in activation space, and that steering hidden states along these subspaces causally controls scoring direction. A linear projection onto bias-direction features predicts judge failures on unseen benchmarks, substantially outperforming text-based alternatives. The work provides a mechanistic account that unifies geometric structure, causal control, and operational prediction within a single framework.