openskillrisk-benchmarking-agent-safety-when-using-real-world-risky-third-party-skills-7902218e·1 events·first seen Aliases: OpenSkillRisk: Benchmarking Agent Safety When Using Real-World Risky Third-Party Skills
Researchers introduce OpenSkillRisk, a safety benchmark of 263 risky third-party skills drawn from public skill marketplaces, designed to evaluate how well LLM-based agent systems recognize and avoid latent execution-time risks. Experiments across three CLI agent frameworks and thirteen LLMs show no system handles risky skills reliably, with even the safest configurations executing unsafe actions in ~17% of cases. The benchmark identifies three recurring failure patterns: failure to recognize risk, recognizing risk but acting anyway, and over-following skill instructions beyond user intent. Context-dependent and system-level risks prove especially difficult for current agents to handle.