6arXiv cs.CL (Computation and Language)·
Framework for computing rigorous probabilistic safety bounds on LLM harmful output
A new arXiv preprint proposes a framework for computing formally sound probabilistic bounds on the likelihood that an LLM generates harmful output for a given prompt. The method applies Clopper-Pearson confidence intervals to obtain PAC (probably approximately correct) bounds, and introduces an algorithm that uses latent-space features to prioritize exploration of harmful branches in the autoregressive generation tree. The approach is designed to produce valid lower bounds even when true harm probability is very small, enabling statistical certification of LLM safety properties.
Related guides (2)
Related events (8)
Online Safety Monitoring for LLMs via Threshold-Based Risk Control
A new arXiv preprint proposes a real-time safety monitor for LLMs that converts an external verifier signal into an alarm by thresholding, with the threshold calibrated via risk control. The authors evaluate the approach on mathematical reasoning and red-teaming datasets, finding it competitive with more complex sequential hypothesis testing monitors. The work addresses the practical deployment problem of detecting unsafe outputs after alignment training.
PAC-Bayes analysis establishes formal expressivity and alignment floors for prompt-conditioned LLMs
A new arXiv preprint models user-LLM interaction as a bilevel cheap-talk game and derives PAC-Bayes bounds showing two irreducible limitations: an 'expressivity floor' where language's finite channel capacity makes distinct tasks indistinguishable, and an 'objective-misalignment floor' where alignment constraints prevent reaching user-ideal outputs. The authors prove that prompt-conditioned LLMs cannot be universal problem solvers, as correct behavior on certain task families is provably unattainable even with infinite data, optimal training, or model scaling. The work suggests multimodal inputs and external memory as potential mitigations by increasing task-relevant information bandwidth.
Formal framework for valid extractable memorization claims in LLMs
A new arXiv preprint proposes a principled methodology for making valid extractable memorization claims about LLMs, addressing both over- and under-statement problems in prior work. The core contribution is a 'matched comparison' approach that measures generation probabilities of training sequences against comparable non-training sequences to establish a calibrated baseline for predictability. Two formalizations are offered: a conformal test for population-level claims and a census method for single-document claims. Applied to OLMo 2 32B on Wikipedia and Llama 3.1 70B on books, the framework reveals significant false-positive rates in naive extraction studies and supports memorization claims at probability thresholds as low as 1e-27.
Uncertainty-aware decision-making algorithms for LLMs using Bayesian and risk-averse methods
A new arXiv preprint proposes and evaluates uncertainty-aware decision-making algorithms for LLMs grounded in Bayesian decision theory and risk-averse decision making, applied to tutoring and automatic peer review tasks. The authors incorporate conformal prediction to provide formal guarantees over strategy and score outputs. Empirical results show Bayesian methods outperform risk-averse rules, which can degrade to generic outputs under high ambiguity. The work highlights a gap in decision-making algorithm research relative to model training advances.
Theory of partially correlated verifier cascades in LLM harnesses: concave log-odds and reliability ceilings
A new arXiv preprint provides a formal theory of serial verification gates in LLM harnesses when verifiers are partially correlated, closing an open problem noted in the recent Odds Law paper (arXiv:2606.15712). Using a de Finetti latent-variable model for per-instance false-accept rates, the authors derive that cascade reliability grows only polynomially (not exponentially) under correlation, that blind-spot atoms impose hard ceilings on extractable evidence, and that adding more gates can sometimes actively harm reliability. Synthetic experiments show independence-based extrapolation underestimates failure by 20x at k=5 and ~3000x at k=10, with the practical implication that decorrelation (changing model family, modality, or evidence source) is the correct lever, not adding more gates.
LLMs susceptible to cognitive heuristics in code vulnerability detection, enabling black-box attacks
A new arXiv paper presents the first systematic study of cognitive heuristics — halo effect, framing effect, and anchoring — in LLM-based code vulnerability detection. Evaluating eight LLMs across three programming languages, the authors find all models susceptible, with framing causing the largest average shift (33.2%), followed by anchoring (23.5%) and halo (18.4%). Critically, the paper demonstrates a proof-of-concept black-box attack that suppresses up to 97% of previously detected vulnerabilities by manipulating surrounding context without changing the code itself. The findings suggest cognitive susceptibility is a systematic and exploitable property of LLM security tooling.
Benchmarking study finds LLMs fail at counterintuitive probability problems despite strong standard performance
A new arXiv paper evaluates 8 state-of-the-art LLMs on discrete probability problems using two datasets: standard exercises (average accuracy 0.96) and counterintuitive exercises designed to trigger heuristic reasoning (average accuracy 0.59). The authors document token bias causing 20%+ performance drops when canonical problem formulations are disguised, and up to 34% degradation when misleading suggestions are embedded in prompts. The findings argue that current LLMs are not genuine probabilistic reasoners despite their success on advanced math benchmarks.
Multi-agent mediation can reverse LLM safety refusals: compositional safety gap demonstrated with GPT-5.6
A new arXiv paper demonstrates that a high-capability LLM (GPT-5.6) can appear safer when exposed directly to a dangerous objective than when that objective is transformed and relayed through a multi-agent pipeline. In the tested setup, an 'Id' and 'Censor' agent rewrite a manipulative instruction before passing it to a 'Superego' user-facing agent, which then produces advice aligned with the manipulative target despite never seeing the raw objective or its provenance. The authors identify this as a compositional safety gap: multi-stage automated workflows can launder dangerous instructions past safety mechanisms by keeping raw instructions outside the downstream model's context window. The finding has direct implications for agentic deployment architectures where end users and safety evaluators only have endpoint-level visibility.