4arXiv cs.CL (Computation and Language)·
Preregistered replication finds monotonicity-agreement boundary in NLI is a selection artifact, not a population property
A preregistered replication study tests whether non-upward monotonicity operators produce lower label agreement in NLI datasets, a finding from prior work on human label variation (HLV). Using unselected SNLI and MultiNLI development sets rather than the disagreement-selected ChaosNLI resource, all seven confirmatory contrasts return effects in the opposite direction and far below the smallest effect size of interest. The authors conclude the earlier negative boundary is likely a selection artifact of ChaosNLI's construction, not a population-level linguistic property. The result carries a methodological warning for HLV research: claims built on selected re-annotation resources should explicitly condition on their selection criteria.
Related guides (1)
Related events (8)
Formal semantic structure explains little of human label variation in NLI, study finds
A preregistered study measures how much formal semantic structure (monotonicity, logical operators) accounts for human label variation in natural language inference, using 3,113 items from ChaosNLI (SNLI and MNLI subsets). Results show a statistically reliable but small group-level effect: non-upward-monotone hypotheses yield higher annotator disagreement (Cliff's delta = -0.284), but formal profiles explain only 3.3–3.6% of entropy variance and cannot identify high-disagreement items (AUC 0.606). Composition-level contrasts return null results, suggesting formal semantic structure shifts disagreement magnitude slightly but does not change what annotators disagree about.
Semantic vs. Surface Noise in LLM Agents: 68-Cell Measurement Study with Held-Out Validation
This paper documents an empirical phenomenon across 10 LLMs from 7 architecture families: meaning-bearing perturbations (paraphrase, synonym substitution) cause final-answer inconsistency ~19.69 percentage points more often than presentation-level perturbations (formatting, reordering) of comparable severity, across GSM8K, MATH, and HotpotQA benchmarks. The effect is validated on a held-out 11th model (qwen2.5-14B-Instruct) with 1,800 trajectories. Trace-level analysis supports a 'stealth-divergence' picture where semantic perturbations preserve the first action but induce divergence in intermediate reasoning steps, while two prior mechanism claims are explicitly retracted. The study is notable for its honest reporting of stress-test failures and pre-registered replication.
Signal Collapse and Reward Hacking in Checker-Guided RAG for Biomedical QA
This paper investigates why NLI-based claim checkers used as process rewards in RL-trained medical RAG agents succeed or fail during training. The authors find that a checker's output distribution during training—not its held-out accuracy—determines whether it provides useful gradient signal, with LLM log-probability scoring causing near-total signal collapse (97%+ neutral labels) while a calibrated MedNLI classifier avoids this. A key finding is that stronger checkers can trigger reward hacking cascades (ultra-short answers, search avoidance, language collapse), while moderate-signal local classifiers yield better final model quality (+12% BERTScore over zero-shot). The work frames these as boundary conditions for verifier-as-reward systems in RLVR pipelines.
LLMs violate statistical self-consistency: the macro fallacy in population-level estimation
A new arXiv paper introduces a framework for testing whether LLM outputs satisfy the law of total probability — specifically, whether prior-weighted conditional estimates over subpopulations aggregate correctly into population-level marginals. Using binary tree partitions and persona prompting across frontier models, the authors find widespread violations of this self-consistency principle. A key finding, termed the 'macro fallacy,' is that estimates reconstructed from fine-grained subpopulation prompts are often more accurate than direct population-level estimates, suggesting models hold relevant subpopulation knowledge but fail to propagate it upward. The work proposes statistical self-consistency as a reference-free, unsaturated evaluation criterion for LLMs.
Resolution Diagnostics for Paired LLM Evaluation: Many Leaderboard Rankings Statistically Unresolved
This paper frames pairwise LLM evaluation as a hypothesis-testing problem and introduces a resolution ratio q = N/N* to diagnose whether leaderboard comparisons are statistically powered. Applying this to two public leaderboards, the authors find that 11/40 Open LLM Leaderboard v1 pairwise comparisons and 4-6/9 MMLU-Pro top-10 adjacent-rank pairs fail to meet conventional (alpha=0.05, power=0.8) resolution targets. A key finding is that the widely-used unpaired Cohen-h shortcut underestimates required sample size by approximately a factor of two in close-comparison regimes, a flaw silently inherited by three major statistical calculators. The unresolved-pair pattern persists under multiplicity correction and sequential testing.
LLMs automate reproducibility assessments in social and behavioral sciences, outperforming human reanalysts
A preprint from arXiv demonstrates that an LLM pipeline can automate reproducibility assessments of published social and behavioral science studies, recovering original effect sizes in 41% of cases (vs. 34% for human reanalysts) and reaching the same qualitative conclusion in 96% of cases (vs. 74% for humans). The study evaluated 76 published studies with predefined claims. The results suggest LLMs could serve as a scalable tool for systematic auditing of empirical research, addressing the resource-intensive nature of traditional reproducibility efforts.
Structural test oracle problem in LLM-as-judge synthetic corpora: silent failures and a validation protocol
A new arXiv paper reports a case where a shared decoding-budget parameter silently truncated hallucinated answers in a multilingual LLM-as-judge evaluation corpus, producing a spurious 32-point cross-lingual accuracy collapse that replicated robustly across sample sizes but was entirely artifactual. The authors argue this failure mode is structural to LLM-generated negative examples, which lack any mechanical item-level integrity check (the 'test oracle problem'), unlike corpora built from deterministic perturbation of gold answers. A second real bias (Markdown formatting preference) was simultaneously distorted in magnitude and sign by the same fault, illustrating that aggregate statistics cannot distinguish fabricated from distorted effects. The paper closes with a validation protocol for analysts working with oracle-less multilingual LLM-as-judge corpora.
Quantifying Cross-Linguistic Effects of Syncretism on Agreement Attraction Using LLM Processing Proxies
This paper investigates why morphological syncretism amplifies agreement attraction errors in some languages (English, German, Russian) but not others (Turkish, Armenian), a pattern lacking a principled account. The authors use surprisal and attention entropy derived from large language models as proxies for human sentence processing across four languages. LLM-derived measures successfully replicate behavioral findings in English and German, align with Turkish null results, and partially capture Russian patterns. The work demonstrates LLMs as tools for cross-linguistic psycholinguistic investigation.