Coefficient of Variation (CV)

Monitoring Agentic Systems Before They're Reliable: A Maturity-Staged Methodology

This paper presents a monitoring and triage methodology for agentic systems in early production, arguing that structural defects—not task-level errors—dominate failure modes at low maturity. The authors decompose evaluation into three dimensions (quality, suitability, efficiency) across three monitoring scopes (within-run, cross-run, structural), using coefficient of variation as a characterization signal and FMEA-adapted severity classification to route findings. Evaluated on a synthetic testbed of 220 runs with controlled error injection, they find that injected task-level errors are indistinguishable from clean baselines when structural defects are present, and that 97% of findings can be routed to automated tracking. They propose a maturity-staging model in which monitoring transitions from structural characterization to error detection to reliability tracking as integration defects resolve.