IBM Research
ibm-research-3e0c08ff·6 events·first seen 1mo agoAliases: IBM Research
Co-occurring entities
More like this (12)
Recent events (6)
IBM and UC Berkeley Diagnose Why Enterprise Agents Fail Using IT-Bench and MAST
IBM Research and UC Berkeley have released IT-Bench and MAST, a benchmark suite and diagnostic framework aimed at evaluating why AI agents fail in enterprise IT environments. The work targets realistic IT operations tasks such as incident response, service management, and infrastructure automation. By categorizing failure modes systematically, MAST provides a structured taxonomy for understanding agent shortcomings beyond simple pass/fail metrics. This addresses a gap in enterprise-focused agent evaluation, where general benchmarks often fail to capture domain-specific complexity.
CUGA on Hugging Face: Democratizing Configurable AI Agents
IBM Research has released CUGA (Configurable Universal Generative Agent) on Hugging Face, positioning it as a framework for building configurable AI agents. The announcement appears on the Hugging Face blog as a tier-2 commentary piece from IBM Research. Details on architecture, benchmarks, and specific capabilities are not available from the body text provided.
The Open Agent Leaderboard
IBM Research and Hugging Face have launched the Open Agent Leaderboard, a public benchmark for evaluating AI agents across standardized tasks. The leaderboard aims to provide transparent, reproducible comparisons of open and proprietary agent systems. This initiative addresses the growing need for rigorous evaluation infrastructure as the agent ecosystem matures.
Inside VAKRA: Reasoning, Tool Use, and Failure Modes of Agents
IBM Research presents an analysis of VAKRA, a benchmark designed to evaluate agentic AI systems on reasoning and tool use capabilities. The post examines how agents fail across different task categories, surfacing systematic failure modes in multi-step reasoning and tool invocation. The analysis provides diagnostic insights into where current agent architectures break down under realistic task conditions.
AssetOpsBench: Bridging the Gap Between AI Agent Benchmarks and Industrial Reality
IBM Research introduces AssetOpsBench, a benchmark designed to evaluate AI agents on industrial asset operations tasks, hosted on Hugging Face. The benchmark targets the gap between existing general-purpose agent benchmarks and real-world industrial deployment scenarios. It provides a playground environment for testing agent capabilities in enterprise/industrial contexts.
ITBench-AA: Frontier Models Score Below 50% on the First Benchmark for Agentic Enterprise IT Tasks
IBM Research and Artificial Analysis have released ITBench-AA, a benchmark targeting agentic AI performance on enterprise IT operations tasks. Frontier models evaluated on the benchmark score below 50%, indicating significant capability gaps in real-world IT automation scenarios. The benchmark appears to be the first of its kind focused specifically on agentic enterprise IT workflows, covering tasks relevant to site reliability engineering and IT operations.