Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents

Cost-aware evaluation framework for offensive and defensive security agents reveals distinct scaling regimes

A new arXiv paper introduces a cost-success evaluation lens for language-model security agents, measuring performance at fixed inference and tool-use budgets rather than peak capability alone. The authors evaluate models on offensive Cybench CTF challenges and defensive Splunk BOTS v1 SOC investigation tasks, finding that offensive performance scales with test-time compute while defensive SOC tasks depend more on disciplined tool use than raw reasoning budget. Scaled open-weight models can approach frontier proprietary systems on offensive tasks while remaining cost-competitive, but defensive agents show a different and harder-to-scale capability profile. The paper argues that security-agent benchmarks should incorporate economic efficiency and operational fit as first-class metrics.