6arXiv cs.CL (Computation and Language)·
OpenSkillRisk benchmark reveals widespread agent safety failures with third-party skills
Researchers introduce OpenSkillRisk, a safety benchmark of 263 risky third-party skills drawn from public skill marketplaces, designed to evaluate how well LLM-based agent systems recognize and avoid latent execution-time risks. Experiments across three CLI agent frameworks and thirteen LLMs show no system handles risky skills reliably, with even the safest configurations executing unsafe actions in ~17% of cases. The benchmark identifies three recurring failure patterns: failure to recognize risk, recognizing risk but acting anyway, and over-following skill instructions beyond user intent. Context-dependent and system-level risks prove especially difficult for current agents to handle.
Related guides (3)
Related events (8)
SkillHarm: Lifecycle-Aware Benchmark for Skill-Based Attacks on AI Agents
SkillHarm is a new benchmark evaluating adversarial attacks on AI agent skills across their full use lifecycle, covering two attack scenarios: Fixed-Payload Poisoning (FPP) and Self-Mutating Poisoning (SMP). The benchmark includes 879 attack samples across 71 skills, organized under a 12-category risk taxonomy targeting data pipelines, system environments, and agent autonomy. Experiments show current agents remain highly vulnerable, with attack success rates up to 86.3% (FPP) and 69.3% (SMP). An automated construction pipeline called AutoSkillHarm, driven by coding agents, was used to generate the benchmark at scale.
ToolBench-X benchmarks LLM agents under tool-environment unreliability
A new arXiv preprint introduces ToolBench-X, a benchmark for evaluating LLM agents under five structured hazard types including Specification Drift, Invocation Error, Execution Failure, Output Drift, and Cross-source Conflict. Each injected hazard remains solvable via recovery paths such as retrying, fallback, or cross-checking, enabling measurement of agent resilience rather than just function-call accuracy. Experiments reveal a substantial reliability gap: agents that perform well in clean environments frequently fail under recoverable hazards, with failures driven by poor hazard diagnosis rather than insufficient tool-use volume or inference budget. The findings argue for shifting tool-use evaluation toward task completion under realistic, unreliable conditions.
SkillGenBench: Benchmarking Skill Generation Pipelines for LLM Agents
SkillGenBench is a new benchmark designed to evaluate the ability of LLM agents to generate correct, reusable, and executable skills from raw repositories and documents, rather than merely using pre-provided skills. It covers two generation regimes (task-conditioned and task-agnostic) and two procedural sources (repository-grounded and document-grounded), with standardized execution-based evaluation protocols. Experiments across multiple skill-generation methods reveal substantial performance variation and distinct failure modes depending on source type. The benchmark aims to establish skill generation as an independent research problem within agent systems.
SkillFuzz: Execution-free fuzzing for implicit intent discovery in LLM agent skill marketplaces
SkillFuzz is a new testing framework that treats skill composition in LLM-based agent marketplaces as a fuzzing problem, using contract-guided Monte Carlo Tree Search to discover 'implicit intents' — unintended behaviors that emerge only when multiple individually-benign skills are co-activated. The approach is execution-free at audit time, relying on structured skill contracts and a skill-free planning baseline as a differential oracle. Across benchmark workloads, it discovers over 1,000 distinct implicit intents within a fixed query budget and confirms more than 80% of highest-risk flagged compositions during execution-time validation.
Biased LLM judges silently disable skill retirement in self-evolving agents
A new arXiv paper analyzes how biased LLM judges used as reward signals in self-evolving agent systems can silently disable the 'skill retirement' mechanism that prevents skill libraries from degrading below a no-skill baseline. The authors show that false-pass bias—where failures are incorrectly scored as passes—disables contribution-based retirement past a sharp threshold that cannot be overcome with more data, while leaving aggregate performance metrics unchanged, making the failure invisible. The paper proposes a defect-injection audit to determine pre-deployment whether a judge falls above or below the critical threshold.
The Regression Tax: Why Adding Skills to LLM Agents Can Make Them Worse
A new arXiv paper introduces the concept of the 'regression tax' — the phenomenon where adding procedural skills to LLM agents causes previously-solved tasks to fail, even when the skill is never invoked. Across nearly 6,000 runs on two office automation benchmarks and three model harness stacks, the authors identify three regression mechanisms: skill description osmosis (context pollution), grounding displacement (procedure overrides input interpretation), and verification displacement (procedure suppresses output checking). The key finding is that top-performing skill configurations succeed primarily by regressing less, not by gaining more, and that grounding and verification are more important to reliability than procedural skill choice.
SkillCenter: 216K-skill open library for grounded autonomous AI agents
SkillCenter is a new open skill library for autonomous AI agents containing 216,938 structured skills across 24 domain bundles, claimed as the largest by total count. The library combines 114,565 source-grounded skills derived from peer-reviewed journals and ArXiv via an LLM-based quality gate (SkillGate), plus 102,373 community skills from GitHub and the ClawHub marketplace. Each retained skill claim is traceable to an exact source quotation, and the library ships as offline-searchable SQLite FTS5 bundles. The work addresses a gap in agent operational knowledge — making outputs not just executable but correct, secure, and maintainable.
Boiling the Frog: A Multi-Turn Benchmark for Agentic Safety
Researchers introduce 'Boiling the Frog,' a multi-turn safety benchmark evaluating whether tool-using AI agents in corporate/office settings are susceptible to incremental attacks that begin with benign requests before introducing harmful payloads. The benchmark uses stateful multi-turn evaluation with a three-level operational risk taxonomy grounded in the EU AI Act and its GPAI Code of Practice. Across nine models, aggregate strict attack success rate is 44.4%, ranging from 20.5% for Claude Haiku 4.5 to 92.9% for Gemini 3.1 Flash Lite, with loss-of-control scenarios reaching 93.3% category-level ASR.