6arXiv cs.AI (Artificial Intelligence)·
Theory of partially correlated verifier cascades in LLM harnesses: concave log-odds and reliability ceilings
A new arXiv preprint provides a formal theory of serial verification gates in LLM harnesses when verifiers are partially correlated, closing an open problem noted in the recent Odds Law paper (arXiv:2606.15712). Using a de Finetti latent-variable model for per-instance false-accept rates, the authors derive that cascade reliability grows only polynomially (not exponentially) under correlation, that blind-spot atoms impose hard ceilings on extractable evidence, and that adding more gates can sometimes actively harm reliability. Synthetic experiments show independence-based extrapolation underestimates failure by 20x at k=5 and ~3000x at k=10, with the practical implication that decorrelation (changing model family, modality, or evidence source) is the correct lever, not adding more gates.
Related guides (2)
Related events (8)
LLM-as-a-Verifier: Training-free verification framework scales along granularity, repetition, and criteria decomposition
Researchers introduce LLM-as-a-Verifier, a general-purpose verification framework that treats verification as a new scaling axis for LLMs, computing continuous scores from token logit distributions rather than discrete judge outputs. The framework scales along three dimensions—score granularity, repeated evaluation, and criteria decomposition—and achieves state-of-the-art results on Terminal-Bench V2 (86.5%), SWE-Bench Verified (78.2%), RoboRewardBench (87.4%), and MedAgentBench (73.3%) without requiring additional training. The authors also demonstrate that the framework's fine-grained signals can serve as dense RL feedback, improving sample efficiency for SAC and GRPO on robotics and math benchmarks, and build a Claude Code extension for monitoring agentic systems.
Online Safety Monitoring for LLMs via Threshold-Based Risk Control
A new arXiv preprint proposes a real-time safety monitor for LLMs that converts an external verifier signal into an alarm by thresholding, with the threshold calibrated via risk control. The authors evaluate the approach on mathematical reasoning and red-teaming datasets, finding it competitive with more complex sequential hypothesis testing monitors. The work addresses the practical deployment problem of detecting unsafe outputs after alignment training.
Co-failure ceiling theorem bounds maximum gains from LLM routing, voting, and mixture-of-agents across 67 frontier models
A new arXiv paper introduces the concept of a 'co-failure ceiling' — the rate at which all models in an ensemble fail on the same query — and proves that no routing, voting, or cascade policy can exceed accuracy of (1 - beta) where beta is this all-wrong rate. Empirically evaluated across 67 models from 21 providers, the paper finds that standard pairwise error correlation metrics systematically underprice the co-failure tail by ~2.5x on open-ended mathematics, and that combining models rarely beats the single best model without strong query-level routing signals. The work provides a finite-sample certificate (via Clopper-Pearson bounds) for the maximum achievable gain from multi-model systems before training a router, and identifies answer format rather than subject matter as a key driver of co-failure on GPQA-Diamond.
Bounding Compositional Incoherence in Multi-Component LLM Agents
This paper formalizes a failure mode in multi-component LLM agent systems where individual components are locally probabilistically coherent but their composition violates basic probability axioms. The authors introduce the 'compositional residual' (eps*) as a runtime-computable measure of this incoherence, finding it positive in 33–94% of ensemble cliques across 1,876 tested configurations on a four-LLM panel. A hierarchical Boyle-Dykstra projection is proposed as a deterministic repair, and an anytime-valid e-process enables sequential monitoring. Notably, three intuitive LLM-side mitigations—retrieval, partition-aware prompting, and aggregator-LLM—each fail or regress.
Formal framework for valid extractable memorization claims in LLMs
A new arXiv preprint proposes a principled methodology for making valid extractable memorization claims about LLMs, addressing both over- and under-statement problems in prior work. The core contribution is a 'matched comparison' approach that measures generation probabilities of training sequences against comparable non-training sequences to establish a calibrated baseline for predictability. Two formalizations are offered: a conformal test for population-level claims and a census method for single-document claims. Applied to OLMo 2 32B on Wikipedia and Llama 3.1 70B on books, the framework reveals significant false-positive rates in naive extraction studies and supports memorization claims at probability thresholds as low as 1e-27.
Semi-supervised framework scales LLM reasoning with minimal labeled data via lightweight verifier
A new arXiv preprint proposes a semi-supervised framework for training LLMs to reason with very few labeled examples, using a lightweight classifier to judge the validity of intermediate reasoning traces. An entropy-based confidence threshold filters unreliable pseudo-labels before fine-tuning. Experiments on math reasoning (Orca-Math subset) and visual QA (GQA) show accuracy comparable to using 10-15x more labeled data. The approach reduces dependence on expensive answer-level supervision by turning verification into a data-creation mechanism.
Early abort cascade for LLM agents predicts episode failure from hidden activations, saving ~47% inference compute
A new arXiv paper introduces a method to detect doomed LLM agent episodes early by probing internal hidden-state activations, rather than waiting for observable failure. The approach uses a cascade of calibrated per-round gates with recall budgets, guaranteeing that eventually-successful episodes survive at a user-specified rate. On TextCraft with Qwen-2.5-7B and Llama-3.2-3B, the cascade saves 37–47% of inference compute at a 90% recall target, outperforming behavior-only baselines by roughly 2x. The work provides both a practical deployment mechanism and theoretical guidance on sample complexity for certifying high recall targets.
Benchmarking study finds LLMs fail at counterintuitive probability problems despite strong standard performance
A new arXiv paper evaluates 8 state-of-the-art LLMs on discrete probability problems using two datasets: standard exercises (average accuracy 0.96) and counterintuitive exercises designed to trigger heuristic reasoning (average accuracy 0.59). The authors document token bias causing 20%+ performance drops when canonical problem formulations are disguised, and up to 34% degradation when misleading suggestions are embedded in prompts. The findings argue that current LLMs are not genuine probabilistic reasoners despite their success on advanced math benchmarks.