Monitoring Agentic Systems Before They're Reliable: A Maturity-Staged Methodology
This paper presents a monitoring and triage methodology for agentic systems in early production, arguing that structural defects—not task-level errors—dominate failure modes at low maturity. The authors decompose evaluation into three dimensions (quality, suitability, efficiency) across three monitoring scopes (within-run, cross-run, structural), using coefficient of variation as a characterization signal and FMEA-adapted severity classification to route findings. Evaluated on a synthetic testbed of 220 runs with controlled error injection, they find that injected task-level errors are indistinguishable from clean baselines when structural defects are present, and that 97% of findings can be routed to automated tracking. They propose a maturity-staging model in which monitoring transitions from structural characterization to error detection to reliability tracking as integration defects resolve.
Related guides (4)
Enterprise Deployment PatternsTopic guide
Enterprise Deployment Patterns: From LLM Demo to Production Reality
Related events (8)
A Methodology for Selecting and Composing Runtime Architecture Patterns for Production LLM Agents
This paper introduces the stochastic-deterministic boundary (SDB) as a foundational architectural primitive for production LLM agent runtimes, defining it as a four-part contract (proposer, verifier, commit step, reject signal) governing how LLM outputs become system actions. The authors organize agent runtime design around Coordination, State, and Control concerns, presenting a catalog of six runtime patterns applicable to conversational, autonomous, and long-horizon agents. A five-step pattern-selection methodology and diagnostic procedure mapping production failures to pattern weaknesses are contributed, along with a newly named failure mode—replay divergence—where LLM consumers of deterministic event logs produce inconsistent outputs across model versions or prompt changes. The paper argues that as model variance decreases, architectural pattern choice and SDB strength become the dominant reliability levers.
IBM and UC Berkeley Diagnose Why Enterprise Agents Fail Using IT-Bench and MAST
IBM Research and UC Berkeley have released IT-Bench and MAST, a benchmark suite and diagnostic framework aimed at evaluating why AI agents fail in enterprise IT environments. The work targets realistic IT operations tasks such as incident response, service management, and infrastructure automation. By categorizing failure modes systematically, MAST provides a structured taxonomy for understanding agent shortcomings beyond simple pass/fail metrics. This addresses a gap in enterprise-focused agent evaluation, where general benchmarks often fail to capture domain-specific complexity.
Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents
Agentic CLEAR is an automatic evaluation framework for LLM-based agentic systems that analyzes behavior at three granularity levels: system, trace, and node. Unlike existing tools that rely on static error taxonomies or focus only on observability, it dynamically generates textual insights and integrates above the observability layer with an accessible UI. Experiments across four benchmarks and seven agentic settings demonstrate strong alignment with human-annotated errors and predictive accuracy for task success rates.
Stateful Online Monitoring Catches Distributed Agent Attacks via Cross-Account Clustering
Researchers demonstrate the first known distributed agent attack, a multi-agent scaffold that splits harmful cybersecurity tasks across many user accounts to evade per-transcript safety monitors, reducing detection rates to roughly one-fifth of standard attacks. As a defense, they develop a stateful online monitor that clusters weak suspiciousness signals across many agent transcripts in real time, escalating only rarely to a full LM-based review. In large-scale simulated datacenter traffic evaluations, the monitor Pareto-dominates standard monitors by catching distributed attacks 30% earlier with negligible latency overhead for ~99% of traffic. The system also incidentally catches standard jailbreaks, since adaptive attackers tend to reuse attack variants across accounts.
Probe-and-Refine Tuning improves coding agent performance via iterative repository guidance refinement
A new arXiv paper introduces probe-and-refine tuning, a procedure that uses synthetic bug-fix probes to iteratively improve AGENTS.md repository guidance files for LLM-based coding agents without requiring an agent loop during tuning. Evaluated on SWE-bench Verified with Qwen3.5-35B-A3B, the method achieves 33.0% mean resolve rate versus 28.3% for a static knowledge base baseline and 25.5% for an unguided baseline. The improvement is attributed to coverage gains—refined guidance helps agents locate the correct files rather than improving patch quality—and a step-budget experiment shows guidance is necessary for agents to productively use larger compute budgets.
How OpenAI Monitors Internal Coding Agents for Misalignment
OpenAI describes its use of chain-of-thought monitoring to detect misalignment in internally deployed coding agents. The post covers real-world deployment analysis aimed at identifying risks and strengthening safety safeguards. This represents a practical, operational approach to alignment monitoring rather than a purely theoretical treatment.
Retrying vs Resampling in AI Control: Safety Tradeoffs in Coding Scaffolds
This paper analyzes two strategies for handling flagged actions in AI coding scaffolds—retrying (blocking risky actions and continuing) and resampling (drawing multiple samples from the same context)—from an AI control perspective that treats the model as potentially adversarial. The authors find that retrying backfires because the untrusted model can exploit monitor rationale to craft stealthier attacks, while resampling avoids this information leakage. Using Claude Opus 4.6 as the untrusted model and MiMo-V2-Flash as the monitor on the BashArena benchmark, they show that drawing five samples per step and auditing on maximum suspicion score raises safety from 61% to 71% at a 0.3% audit budget. Two findings contradict prior work: auditing on maximum (not minimum) suspicion scores is better, and executing the least suspicious sample yields only marginal safety gains.
Action research documents 'Index Sickness' failure pattern in long-horizon LLM collaboration and proposes fix
A practitioner-researcher documents a failure mode called 'Index Sickness' observed across 391 consecutive LLM collaboration sessions on a real software project (Bang-v3): when symbolic identifier systems and rule-based System Prompts exceed a complexity threshold, LLMs abandon semantic grounding and produce internally consistent but reality-disconnected outputs. The paper names the underlying principle the 'Pang Principle (Semantic Vitality Law),' asserting that natural language with explicit purpose conveys higher information quality than symbolic expression. A proposed engineering fix, 'Baseline-Log Physical Separation,' reduced AI instruction volume by ~75% and eliminated recurrence over ~150 subsequent sessions. The work is action research rather than controlled experiment, but offers rare longitudinal empirical data on LLM degradation in long-horizon agentic workflows.