6arXiv cs.CL (Computation and Language)·
Mechanistic interpretability study reveals geometric structure of LLM-as-judge bias in hidden states
A new arXiv preprint investigates LLM-as-judge scoring bias at the representation level rather than the input-output level, studying seven judge models across seven bias types and nine benchmarks. The authors find that biased inputs are displaced along low-dimensional, type-specific subspaces in activation space, and that steering hidden states along these subspaces causally controls scoring direction. A linear projection onto bias-direction features predicts judge failures on unseen benchmarks, substantially outperforming text-based alternatives. The work provides a mechanistic account that unifies geometric structure, causal control, and operational prediction within a single framework.
Related guides (2)
Related events (8)
Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modeling
This paper identifies and analyzes 'Perceptual Judgment Bias' in multimodal LLM judges, where models anchor on response text rather than visual evidence when the two conflict. The authors introduce a Perceptually Perturbed Judgment Dataset using counterfactual responses to isolate perceptual errors, and a training framework combining GRPO-based reward modeling with batch-ranking objectives. Experiments on MLLM-as-a-Judge benchmarks show improved perceptual fidelity, ranking coherence, and alignment with human evaluation.
Auditing LLM-as-Judge reliability: judge upgrades are not interchangeable across model families
A new arXiv paper investigates measurement validity problems in LLM-as-judge evaluation, finding that swapping evaluator models changes scores even when candidate responses are fixed. Across four judgment datasets, the authors compare Qwen3 dense judges (1.7B–32B) and MiniMax M2/M2.7 API releases, finding that only the Qwen3 1.7B→4B upgrade yields robust adjacent gains while MiniMax adjacent releases do not. Stronger judges reduce but do not eliminate position and verbosity bias, and repeated-sample juries add little when errors are correlated. The paper argues for standardized reporting requirements including dataset slices, bias probes, error-dependence estimates, and protocol audit trails.
Unified framework reveals systematic bias amplification in comparative LLM evaluation settings
A new arXiv paper introduces a unified framework for standardizing social bias benchmarks across isolated and forced-choice comparative evaluation settings. The study finds a large 'paradigm gap': comparative settings act as aggressive catalysts for latent discrimination compared to isolated assessments, and Chain-of-Thought reasoning exacerbates this effect rather than mitigating it. Critically, this comparative bias persists even when models are given neutral fallback options or claim to answer randomly, and scales positively with model size. The authors recommend comparative settings for auditing but warn practitioners against using comparative deployments in ambiguous real-world tasks.
Mechanistic analysis of how LLMs encode essay quality in internal representations
Researchers systematically probe the hidden representations of eight LLMs across three essay datasets (ASAP++, CSEE, ENEM) to understand how automated essay scoring (AES) works internally. Using linear probing, dimensionality reduction, and neuron-level analysis, they find essay quality is encoded in a linearly accessible form that emerges progressively across layers and partially transfers across prompts. Individual 'essay scoring neurons' are identified whose activations correlate with scores and respond to targeted interventions, with longer essays relying more on deeper layers. The work contributes to mechanistic interpretability of LLM-based scoring systems.
LLMs judge worse than they generate: empirical challenge to self-evaluation pipeline assumptions
A new arXiv preprint tests the implicit assumption that LLM evaluation is easier than generation, using a controlled in-context QA setup across four benchmarks (SQuAD 2.0, DROP, HotpotQA, MuSiQue) and two models. Results show generation accuracy exceeds self-evaluation accuracy on three of four benchmarks, with attention analysis revealing that evaluation attends to context 3–5x less than generation does. LoRA fine-tuning experiments confirm the asymmetry is not a training artifact, with cross-task interference observed in both directions. The findings directly challenge assumptions underlying LLM-as-a-Judge and self-evaluation pipelines widely used in RLHF and agentic systems.
Benchmarking LLM judges for citation verification in deep-research systems
A new arXiv paper evaluates 8 LLM judges from 3 model families on citation quality assessment for deep-research systems, testing across 1,248 rubric decisions with human-reviewed gold labels. The study finds that cheaper models remain competitive with frontier models — GPT-5-mini achieves the strongest source-relevance F1 at 0.908 — but judges differ substantially in directional bias (pass-rate drift, false positive/negative rates) even when scalar F1 scores are similar. The key finding is that scalar F1 obscures biases that would be directly reinforced in an RL training loop, making judge calibration a prerequisite before using citation rubrics as reward signals.
BabelJudge: Benchmark for measuring LLM-as-a-judge reliability across languages and agent trajectories
BabelJudge is a new open-source benchmark and audit framework that systematically measures four failure modes in LLM-as-a-judge systems: position bias, verbosity bias, order inconsistency, and cross-lingual degradation. The framework uses a 'gold-labelling by degradation' technique to generate labeled evaluation pairs without human annotation. Evaluation of Qwen2.5-7B-Instruct-4bit across English, Hindi, Arabic, and Swahili reveals severe cross-lingual reliability drops, with Swahili order consistency collapsing to near-random (0.480). The framework is extended to agentic evaluation with nine trajectory-level perturbations and three new metrics, released as a Python package supporting 11 judge backends.
Biased LLM judges silently disable skill retirement in self-evolving agents
A new arXiv paper analyzes how biased LLM judges used as reward signals in self-evolving agent systems can silently disable the 'skill retirement' mechanism that prevents skill libraries from degrading below a no-skill baseline. The authors show that false-pass bias—where failures are incorrectly scored as passes—disables contribution-based retirement past a sharp threshold that cannot be overcome with more data, while leaving aggregate performance metrics unchanged, making the failure invisible. The paper proposes a defect-injection audit to determine pre-deployment whether a judge falls above or below the critical threshold.