5arXiv cs.CL (Computation and Language)·
OAT: Unsupervised failure attribution for LLM agents via one-class learning on successful trajectories
A new arXiv preprint introduces OAT, a method for attributing failures in LLM-based agentic systems by training exclusively on successful trajectories using one-class learning with neural controlled differential equations. At inference time, each step in a failure trajectory receives an anomaly score based on deviation from learned success dynamics. OAT is reported to be 200–5000× faster than prompting-based baselines while outperforming them by +20% F1 in-domain and +7% out-of-distribution, using only 100 successful trajectories for training. The approach addresses a practical bottleneck in agentic system debugging by eliminating the need for costly step-level failure annotations.
Related guides (2)
Related events (8)
Progress Advantage: Annotation-Free Step-Level Scoring for LLM Agents via RL Post-Training
Researchers introduce 'progress advantage,' a method that derives implicit step-level reward signals for LLM agents directly from the log-probability ratio between an RL-trained policy and its reference policy, without requiring dedicated process reward model training. The approach is shown to recover the optimal advantage function under a general stochastic MDP formulation, making it annotation-free and domain-agnostic. Validated across five benchmarks and four model families on tasks including test-time scaling, uncertainty quantification, and failure attribution, it outperforms confidence-based baselines and even dedicated trained reward models. The result is practically significant because building process reward models for agentic settings is currently a major bottleneck.
Early abort cascade for LLM agents predicts episode failure from hidden activations, saving ~47% inference compute
A new arXiv paper introduces a method to detect doomed LLM agent episodes early by probing internal hidden-state activations, rather than waiting for observable failure. The approach uses a cascade of calibrated per-round gates with recall budgets, guaranteeing that eventually-successful episodes survive at a user-specified rate. On TextCraft with Qwen-2.5-7B and Llama-3.2-3B, the cascade saves 37–47% of inference compute at a 90% recall target, outperforming behavior-only baselines by roughly 2x. The work provides both a practical deployment mechanism and theoretical guidance on sample complexity for certifying high recall targets.
Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents
Agentic CLEAR is an automatic evaluation framework for LLM-based agentic systems that analyzes behavior at three granularity levels: system, trace, and node. Unlike existing tools that rely on static error taxonomies or focus only on observability, it dynamically generates textual insights and integrates above the observability layer with an accessible UI. Experiments across four benchmarks and seven agentic settings demonstrate strong alignment with human-annotated errors and predictive accuracy for task success rates.
Failed reasoning traces encode recoverability structure for test-time routing and post-training analysis
A new arXiv paper argues that failed reasoning traces from post-trained LLMs contain exploitable signal about whether failures are recoverable via resampling or require structural intervention. The authors derive three trajectory features from the distributional signature of failed rollouts (not their text content) that cluster failures into stable regimes and characterize failure topography across post-training methods with 84.3% accuracy. A training-free routing rule built on these features lifts rescue rates by +12.2% on a deployment-relevant hard subset, and the features transfer across model families. The work reframes failed traces as diagnostic objects rather than discarded data, with implications for inference-time compute allocation and post-training analysis.
O-VAD: Training-free agentic framework for industrial video anomaly detection via object-centric tracking
Researchers introduce O-VAD, a training-free agentic framework for Industrial Video Anomaly Detection (IVAD) that tracks spatial-temporal object state trajectories and reasons over them to identify anomalies without domain-specific knowledge or retraining on normal clips. The system is designed to handle the complexity of industrial settings with strict physics and procedural constraints, outperforming frontier VLMs, agentic frameworks, and fine-tuned traditional VAD methods on three IVAD benchmarks. The approach emphasizes interpretable reporting of anomaly processes and types, mimicking human inspector reasoning.
Mechanism-driven internal monitors detect LLM training instability thousands of steps before loss divergence
A new arXiv preprint proposes mechanism-driven monitoring signals derived from the functional roles of critical modules (low-precision flash attention, MoE routers) to detect training instability before it manifests in loss or gradient norms. The authors derive monitors such as spectral entropy of a QK bilinear decomposition and MoE router indicators, showing via fault-injection experiments that these signals trigger thousands of steps ahead of loss divergence. The work targets a high-cost failure mode in frontier LLM training where instability can persist undetected for thousands of steps on expensive accelerator fleets.
STRACE framework improves LLM agent optimization via structural trajectory analysis and causal extraction
Researchers introduce STRACE (Structural TRajectory Analysis and Causal Extraction), a framework for constructing high signal-to-noise optimization contexts for long-horizon LLM agents. At the batch level, STRACE mines failure patterns to filter redundant traces; within each trace, it performs causal localization over a textual dependency graph to isolate root-cause steps. On the formal verification benchmark VeruSAGE-Bench, STRACE achieves a 1.4× success-rate improvement (42.5% to 58.5%) over human-expert-designed agents, outperforming standard context-filtering baselines.
Role-Agent: Bootstrapping LLM Agents via Dual-Role Evolution
Role-Agent is a new framework that uses a single LLM simultaneously as both agent and environment, enabling self-bootstrapped co-evolution without external environment feedback. The system has two components: World-In-Agent (WIA), which uses predicted vs. actual state alignment as a process reward, and Agent-In-World (AIW), which reshapes training data by retrieving tasks with similar failure patterns. Experiments across multiple benchmarks show an average performance gain of over 4% over strong baselines. The approach addresses key limitations in LLM agent training: inefficient feedback and static environments.