Rethinking how we measure AI intelligence
DeepMind has announced Game Arena, a new open-source evaluation platform designed for rigorous head-to-head comparison of frontier AI models. The platform uses environments with clear winning conditions to assess model capabilities. This represents DeepMind's contribution to addressing ongoing concerns about the adequacy of existing AI benchmarks.
Related guides (2)
Related events (8)
Measuring progress toward AGI: A cognitive framework
DeepMind is introducing a cognitive framework designed to measure progress toward AGI, providing structured criteria for assessing how close AI systems are to general intelligence. Alongside the framework, they are launching a Kaggle hackathon to crowdsource the development of relevant evaluations. The announcement signals a formal effort by a Tier 1 lab to operationalize AGI progress measurement, which has historically been contested and informal.
Introducing AI vs. AI: A Deep Reinforcement Learning Multi-Agent Competition System
Hugging Face has launched 'AI vs. AI', a competition framework for evaluating deep reinforcement learning agents through head-to-head multi-agent matchups. The system is designed to benchmark RL agents against each other in competitive environments rather than static benchmarks. This represents a new evaluation paradigm for RL research hosted on the Hugging Face platform.
DeepMind Publishes Framework for Evaluating Cybersecurity Threats of Advanced AI
DeepMind has released a framework designed to help cybersecurity experts assess and prioritize defenses against potential threats posed by advanced AI systems. The framework aims to systematically identify which defensive measures are necessary given AI's expanding capabilities in offensive cyber operations. This represents DeepMind's structured approach to evaluating AI-enabled cyber risks before they materialize at scale.
OpenAI Releases Universe: A Platform for Training AI Across Games, Websites, and Applications
OpenAI released Universe, a software platform designed to measure and train AI general intelligence across a broad range of environments including games, websites, and other applications. The platform aims to expose AI agents to the world's supply of software as training and evaluation environments. This represented an early effort to develop general-purpose AI agents capable of operating across diverse real-world interfaces.
Launching the Artificial Analysis Text to Image Leaderboard & Arena
Hugging Face and Artificial Analysis are launching a combined leaderboard and arena for evaluating text-to-image models. The leaderboard tracks quality, speed, and cost metrics across leading image generation models, while the arena component collects human preference votes for side-by-side comparisons. This provides a structured benchmark for comparing commercial and open-weight image generation systems.
Measuring AI's capability to accelerate biological research
OpenAI introduces a real-world evaluation framework designed to measure how AI systems can accelerate biological research in wet lab settings. The work uses GPT-5 to optimize a molecular cloning protocol as a concrete demonstration case. The framework explicitly addresses both the potential benefits and biosecurity risks of AI-assisted experimentation, positioning this as a dual-use capability assessment.
Introducing HealthBench
OpenAI has released HealthBench, a new evaluation benchmark designed to assess AI model performance and safety in healthcare settings. The benchmark was developed with input from over 250 physicians and targets realistic clinical scenarios. It aims to establish a shared standard for measuring how well AI models handle health-related tasks.
Open-world evaluations for measuring frontier AI capabilities: Introducing CRUX
This commentary introduces CRUX, a new evaluation project designed to assess frontier AI systems on long-horizon, open-ended, and messy real-world tasks. The piece argues that existing benchmarks are insufficient for capturing the full range of capabilities exhibited by frontier models in complex settings. CRUX aims to fill this gap by providing evaluations that better reflect deployment-relevant performance.