5arXiv cs.CL (Computation and Language)·
ESPP: Simulated social persona panels improve LLM-as-judge evaluation of generative UI
Researchers introduce ESPP (Evidence-Grounded, Social-Weighted Persona Panel), a three-stage evaluation method for Generative UI that simulates psychologically diverse personas, has them exchange opinions under a bounded-confidence mechanism, and aggregates judgments via Delphi-inspired social weighting. ESPP raises Pearson correlation with human judgment from 0.716 to 0.922 compared to a naive single-pass LLM judge, with prompt-ensemble baselines recovering only about a third of the gap. The method also reveals that user subgroups agree on overall model rankings but diverge on specific rating dimensions — a structural disagreement a single homogeneous judge would erase.
Related guides (1)
Related events (8)
LLMs judge worse than they generate: empirical challenge to self-evaluation pipeline assumptions
A new arXiv preprint tests the implicit assumption that LLM evaluation is easier than generation, using a controlled in-context QA setup across four benchmarks (SQuAD 2.0, DROP, HotpotQA, MuSiQue) and two models. Results show generation accuracy exceeds self-evaluation accuracy on three of four benchmarks, with attention analysis revealing that evaluation attends to context 3–5x less than generation does. LoRA fine-tuning experiments confirm the asymmetry is not a training artifact, with cross-task interference observed in both directions. The findings directly challenge assumptions underlying LLM-as-a-Judge and self-evaluation pipelines widely used in RLHF and agentic systems.
Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modeling
This paper identifies and analyzes 'Perceptual Judgment Bias' in multimodal LLM judges, where models anchor on response text rather than visual evidence when the two conflict. The authors introduce a Perceptually Perturbed Judgment Dataset using counterfactual responses to isolate perceptual errors, and a training framework combining GRPO-based reward modeling with batch-ranking objectives. Experiments on MLLM-as-a-Judge benchmarks show improved perceptual fidelity, ranking coherence, and alignment with human evaluation.
Auditing LLM-as-Judge reliability: judge upgrades are not interchangeable across model families
A new arXiv paper investigates measurement validity problems in LLM-as-judge evaluation, finding that swapping evaluator models changes scores even when candidate responses are fixed. Across four judgment datasets, the authors compare Qwen3 dense judges (1.7B–32B) and MiniMax M2/M2.7 API releases, finding that only the Qwen3 1.7B→4B upgrade yields robust adjacent gains while MiniMax adjacent releases do not. Stronger judges reduce but do not eliminate position and verbosity bias, and repeated-sample juries add little when errors are correlated. The paper argues for standardized reporting requirements including dataset slices, bias probes, error-dependence estimates, and protocol audit trails.
LLMs fail to consistently simulate demographic perspective-taking in hate speech annotation
A new arXiv paper evaluates whether persona-conditioned LLMs can replicate how different demographic groups perceive hate speech, testing three dimensions: inter-group disagreement, in-group sensitivity, and vicarious prediction. No model consistently captures all three dimensions, and performance is highly model-dependent rather than emerging reliably from identity prompts alone. Vicarious prompting with Llama 3.1 provides the closest approximation to human disagreement patterns across demographic axes. The findings have implications for using LLMs as proxies for diverse human annotators in content moderation tasks.
Expert Support Case Study: Bolstering a RAG App with LLM-as-a-Judge
Hugging Face published a case study describing how Digital Green used an LLM-as-a-Judge approach to evaluate and improve a retrieval-augmented generation (RAG) application. The post covers the methodology for using LLMs to score and validate RAG outputs, providing a practical deployment pattern for quality assurance in production AI systems. It serves as a concrete example of enterprise-grade evaluation pipelines built on top of RAG architectures.
Reason-Mediated Behavioral Models for Auditing LLM Social Simulators
A new arXiv paper proposes an evaluation framework for LLMs used as social simulators, arguing that matching final outcomes is insufficient — simulators must also replicate the underlying rationale-derived reasoning paths. Using a 94-person sunscreen concept test, the authors map open-ended human rationales into signed 'reason states' and show that LLM-simulated reasons, while superficially plausible, frequently mirror the stimulus material rather than recovering the respondent's actual acceptance or rejection logic. The framework provides an interpretable audit for whether LLM social simulators are reasoning correctly, not just producing correct-looking outputs.
Judge Arena: Benchmarking LLMs as Evaluators
Hugging Face and Atla have launched Judge Arena, a platform for benchmarking large language models in their role as automated evaluators. The initiative uses an Elo-based ranking system to compare how well different LLMs judge the quality of model outputs, addressing the growing reliance on LLM-as-judge paradigms in evaluation pipelines. This fills a meta-evaluation gap: as LLM judges become standard practice, understanding their relative reliability and biases becomes critical infrastructure for the field.
Controlled study finds LLM user ratings reflect pre-interaction framing more than actual model performance
A controlled experiment with 162 participants tested six LLMs across three tasks, manipulating whether users were told the model was better or worse than it actually was. Post-interaction ratings were predicted by whether the model met expectations (β≈0.47–0.50) and user confidence, not by task performance (β≈-0.01–0.11, n.s.). Actual output quality depended only on true model capability, not framing. The authors conclude that user-elicited evaluations—including preference data driving public leaderboards—measure expectation management at least as much as model quality.