SynthAVE

SynthAVE: Multi-LLM arena validation achieves near-human agreement for synthetic e-commerce labeling at scale

Researchers introduce SynthAVE, a benchmark and methodology for scalable synthetic label generation for e-commerce attribute value extraction, covering 12,726 products across 229 product types, 792 attributes, and 4 languages. The core contribution is a multi-LLM arena validation framework using 21 judge configurations (7 model families × 3 prompts) with majority voting, achieving Cohen's κ=0.92 agreement with human experts. The approach addresses the prohibitive cost of human annotation at industrial scale while maintaining quality parity, demonstrating that diverse model ensembles aggregate into highly reliable predictions even when individual judges disagree.