Krippendorff's alpha

Action-graded severity scale for red-teaming tool-using AI agents beyond binary attack-success rate

A new arXiv paper argues that binary attack-success rate metrics for agentic red-teaming discard critical defender-relevant information about harm severity. The authors introduce a seven-level ordinal harm rubric (L0–L6) grading an agent's tool-call trajectory by reversibility, scope crossing, and privilege escalation, computed via both a deterministic oracle and a three-model LLM judge panel. Applied to four victim models and two defenses on the AgentDojo benchmark suite, the rubric exposes cases the binary metric misses—including a defense reporting zero attack-success rate that still permits cross-scope data leaks. The judge panel achieves high ordinal agreement (Krippendorff's alpha = 0.91) but shares systematic blind spots around escalation chain recognition.