5arXiv cs.CL (Computation and Language)·
SynthAVE: Multi-LLM arena validation achieves near-human agreement for synthetic e-commerce labeling at scale
Researchers introduce SynthAVE, a benchmark and methodology for scalable synthetic label generation for e-commerce attribute value extraction, covering 12,726 products across 229 product types, 792 attributes, and 4 languages. The core contribution is a multi-LLM arena validation framework using 21 judge configurations (7 model families × 3 prompts) with majority voting, achieving Cohen's κ=0.92 agreement with human experts. The approach addresses the prohibitive cost of human annotation at industrial scale while maintaining quality parity, demonstrating that diverse model ensembles aggregate into highly reliable predictions even when individual judges disagree.
Related guides (2)
Related events (8)
Judge Arena: Benchmarking LLMs as Evaluators
Hugging Face and Atla have launched Judge Arena, a platform for benchmarking large language models in their role as automated evaluators. The initiative uses an Elo-based ranking system to compare how well different LLMs judge the quality of model outputs, addressing the growing reliance on LLM-as-judge paradigms in evaluation pipelines. This fills a meta-evaluation gap: as LLM judges become standard practice, understanding their relative reliability and biases becomes critical infrastructure for the field.
LLM-as-a-Verifier: Training-free verification framework scales along granularity, repetition, and criteria decomposition
Researchers introduce LLM-as-a-Verifier, a general-purpose verification framework that treats verification as a new scaling axis for LLMs, computing continuous scores from token logit distributions rather than discrete judge outputs. The framework scales along three dimensions—score granularity, repeated evaluation, and criteria decomposition—and achieves state-of-the-art results on Terminal-Bench V2 (86.5%), SWE-Bench Verified (78.2%), RoboRewardBench (87.4%), and MedAgentBench (73.3%) without requiring additional training. The authors also demonstrate that the framework's fine-grained signals can serve as dense RL feedback, improving sample efficiency for SAC and GRPO on robotics and math benchmarks, and build a Claude Code extension for monitoring agentic systems.
AI-PAVE-Br: LLM-based product attribute extraction system and Portuguese benchmark dataset for Brazilian e-commerce
Researchers introduce AI-PAVE-Br, an LLM-based system for Product Attribute Value Extraction (PAVE) tailored to Brazilian e-commerce catalogs in Portuguese. The paper also releases the Golden Set, a manually annotated benchmark dataset for PAVE in Portuguese, structured with entity, category, and subcategory annotations. Experiments show AI-PAVE-Br with prompt engineering substantially outperforms conventional NER baselines. The work addresses a gap in non-English NLP resources for structured e-commerce data extraction.
WhoSaidIt: Human-LLM Collaborative Annotation for Multilingual Speaker-Attribute Classification
This paper proposes a human-LLM collaborative re-annotation framework for stabilizing noisy multilingual speaker-attribute labels under resource constraints. LLMs surface recurring annotation rationales through iterative expert interaction, combined with disagreement-focused sampling for targeted re-annotation. The resulting WhoSaidIt dataset covers nine speaker-attribute labels across multiple languages. Benchmarking of recent LLMs reveals substantial cross-lingual annotation divergence and highlights both capabilities and limitations of LLMs in this classification task.
SynAE: Framework for Evaluating Synthetic Data Quality in Tool-Calling Agent Benchmarks
SynAE is a proposed evaluation framework for measuring how well synthetic datasets replicate and augment real data trajectories for multi-turn, tool-calling agent testing. It assesses validity, fidelity, and diversity across four metric categories: task instructions, tool calls, final outputs, and downstream evaluation. The paper demonstrates that no single metric suffices to characterize synthetic data quality, motivating multi-axis evaluation. A demo and code are publicly available.
Benchmarking LLM judges for citation verification in deep-research systems
A new arXiv paper evaluates 8 LLM judges from 3 model families on citation quality assessment for deep-research systems, testing across 1,248 rubric decisions with human-reviewed gold labels. The study finds that cheaper models remain competitive with frontier models — GPT-5-mini achieves the strongest source-relevance F1 at 0.908 — but judges differ substantially in directional bias (pass-rate drift, false positive/negative rates) even when scalar F1 scores are similar. The key finding is that scalar F1 obscures biases that would be directly reinforced in an RL training loop, making judge calibration a prerequisite before using citation rubrics as reward signals.
Calibrated LLM annotation and encoder transfer for measuring human values in social media text
A new arXiv preprint investigates how different LLMs, prompts, and instruction languages operationalize Schwartz's theory of basic human values when annotating non-English social media posts. The authors evaluate annotation quality beyond standard F1 metrics, examining structural alignment, error structure, and confidence-ambiguity relations, finding that iterative prompt calibration reduces misattributions. They also demonstrate that LLM annotations can be transferred to a smaller encoder model via soft-label training, preserving theory-grounded value interpretations and uncertainty information.
LLMs judge worse than they generate: empirical challenge to self-evaluation pipeline assumptions
A new arXiv preprint tests the implicit assumption that LLM evaluation is easier than generation, using a controlled in-context QA setup across four benchmarks (SQuAD 2.0, DROP, HotpotQA, MuSiQue) and two models. Results show generation accuracy exceeds self-evaluation accuracy on three of four benchmarks, with attention analysis revealing that evaluation attends to context 3–5x less than generation does. LoRA fine-tuning experiments confirm the asymmetry is not a training artifact, with cross-task interference observed in both directions. The findings directly challenge assumptions underlying LLM-as-a-Judge and self-evaluation pipelines widely used in RLHF and agentic systems.