Entity · paper

The Test Oracle Problem in Synthetic LLM-as-Judge Corpora: Disappearance, Distortion and a Validation Protocol

paperactiveprovisionalthe-test-oracle-problem-in-synthetic-llm-as-judge-corpora-disappearance-distortion-and-a-validation-protocol-78d1e16b·1 events·first seen 3h ago

Aliases: The Test Oracle Problem in Synthetic LLM-as-Judge Corpora: Disappearance, Distortion and a Validation Protocol

More like this (12)

Can LLMs Judge Better Than They Generate? Evaluating Task Asymmetry, Mechanistic Interpretability and Transferability for In-Context QA When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias Auditing Protocol-Level Shortcuts in Large Audio Language Model Judges for Speech Evaluation The Anatomy of the CTC Oracle Gap: Acoustic Exhaustion and Linguistic Recovery A Komi-Yazva–Russian Parallel Corpus and Evaluation Protocol for Zero- and Few-Shot LLM Translation Challenges and Recommendations for LLMs-as-a-Judge in Multilingual Settings and Low-Resource Languages Validity of LLMs as data annotators: AMALIA on authority Accuracy and Satisfaction in Multi-Turn LLM Dialogues for NFR Assessment Flaws in the LLM Automation Narrative Operadic consistency: a label-free signal for compositional reasoning failures in LLMs Revising Context, Shifting Simulated Stance: Auditing LLM-Based Stance Simulation in Online Discussions

Recent events (1)

6arXiv · cs.CL·3h ago·source ↗

Structural test oracle problem in LLM-as-judge synthetic corpora: silent failures and a validation protocol

A new arXiv paper reports a case where a shared decoding-budget parameter silently truncated hallucinated answers in a multilingual LLM-as-judge evaluation corpus, producing a spurious 32-point cross-lingual accuracy collapse that replicated robustly across sample sizes but was entirely artifactual. The authors argue this failure mode is structural to LLM-generated negative examples, which lack any mechanical item-level integrity check (the 'test oracle problem'), unlike corpora built from deterministic perturbation of gold answers. A second real bias (Markdown formatting preference) was simultaneously distorted in magnitude and sign by the same fault, illustrating that aggregate statistics cannot distinguish fabricated from distorted effects. The paper closes with a validation protocol for analysts working with oracle-less multilingual LLM-as-judge corpora.

Evaluation and Benchmarking AI Safety Research The Test Oracle Problem in Synthetic LLM-as-Judge Corpora: Disappearance, Distortion and a Validation Protocol