EVA-Bench Data 2.0: Expanded agentic tool-use evaluation benchmark with 121 tools and 213 scenarios
ServiceNow AI has released EVA-Bench Data 2.0, an evaluation benchmark covering 3 domains, 121 tools, and 213 scenarios for assessing agentic AI systems. The benchmark appears designed to measure tool-use and multi-step task completion capabilities across diverse enterprise-relevant contexts. This expands the evaluation surface for agent benchmarking, which remains an active area of development.
Related guides (2)
Related events (8)
A New Framework for Evaluating Voice Agents (EVA)
ServiceNow AI has published a blog post on Hugging Face introducing EVA, a new evaluation framework designed specifically for voice agents. The framework appears to address gaps in existing evaluation methodologies for assessing voice-based AI agent performance. As voice agents become more prevalent in enterprise and consumer settings, standardized evaluation protocols are increasingly important for benchmarking progress.
T1-Bench: Multi-scenario agent benchmark across 25 real-world domains
T1-Bench is a new benchmark for evaluating agentic LLM systems in realistic customer-facing, multi-domain environments, covering 25 domains of varying difficulty with interleaved multi-turn scenarios. The authors evaluate 12 proprietary and open-weight models and combine automatic evaluation with human judgments. The benchmark targets gaps in existing agent evals around task complexity, domain diversity, and compositional reasoning across multi-step interactions.
ITBench-AA: Frontier Models Score Below 50% on the First Benchmark for Agentic Enterprise IT Tasks
IBM Research and Artificial Analysis have released ITBench-AA, a benchmark targeting agentic AI performance on enterprise IT operations tasks. Frontier models evaluated on the benchmark score below 50%, indicating significant capability gaps in real-world IT automation scenarios. The benchmark appears to be the first of its kind focused specifically on agentic enterprise IT workflows, covering tasks relevant to site reliability engineering and IT operations.
OpenEnv in Practice: Evaluating Tool-Using Agents in Real-World Environments
This Hugging Face blog post introduces OpenEnv, a framework for evaluating tool-using AI agents in real-world environments. The piece appears to address the challenge of benchmarking agentic systems that interact with external tools and environments, moving beyond static benchmarks toward dynamic, practical evaluation settings. As a tier-2 commentary piece, it likely discusses methodology, design choices, and results from applying OpenEnv to assess agent capabilities.
AssetOpsBench: Bridging the Gap Between AI Agent Benchmarks and Industrial Reality
IBM Research introduces AssetOpsBench, a benchmark designed to evaluate AI agents on industrial asset operations tasks, hosted on Hugging Face. The benchmark targets the gap between existing general-purpose agent benchmarks and real-world industrial deployment scenarios. It provides a playground environment for testing agent capabilities in enterprise/industrial contexts.
Introducing EVMbench: AI Agent Benchmark for Smart Contract Vulnerabilities
OpenAI and Paradigm have jointly introduced EVMbench, a benchmark designed to evaluate AI agents on their ability to detect, patch, and exploit high-severity vulnerabilities in Ethereum Virtual Machine (EVM) smart contracts. The benchmark targets a specialized security domain requiring both code understanding and adversarial reasoning. This represents a new evaluation surface for frontier AI agents in the context of blockchain security.
Automated Benchmark Auditing for AI Agents and Large Language Models (ABA)
The paper introduces Auto Benchmark Audit (ABA), an agentic framework that systematically audits AI benchmark tasks for issues such as ambiguous specifications, environment conflicts, and incorrect ground truths. Applied to 168 benchmarks across nine domains including NeurIPS publications, ABA identifies critical issues in over 25.7% of evaluated tasks. The authors demonstrate that filtering out flawed tasks materially shifts model rankings and improves average performance on SWE-bench Verified and Terminal-Bench 2 by 9.9% and 9.6% respectively, indicating that current benchmark scores are significantly distorted by task quality problems. The agentic tool and annotations are released publicly.
Introducing HealthBench
OpenAI has released HealthBench, a new evaluation benchmark designed to assess AI model performance and safety in healthcare settings. The benchmark was developed with input from over 250 physicians and targets realistic clinical scenarios. It aims to establish a shared standard for measuring how well AI models handle health-related tasks.