OpenAI Introduces IndQA: Multilingual Benchmark for Indian Languages
OpenAI has released IndQA, a benchmark designed to evaluate AI systems across 12 Indian languages and 10 knowledge domains. The benchmark was developed with domain experts and focuses on cultural understanding and reasoning capabilities. It targets a significant gap in multilingual evaluation coverage for South Asian languages.
Related guides (3)
Related events (8)
IndicContextEval: Benchmark for context utilisation in Audio LLMs across 8 Indic languages
Researchers introduce IndicContextEval, a 56-hour multilingual speech benchmark covering 555 speakers across 8 Indian languages and 23 professional domains, designed to test whether Audio LLMs genuinely use textual context (domain descriptions, entity lists) or rely on parametric knowledge. The benchmark employs a 7-level prompting framework that progressively introduces contextual signals including adversarial prompts with incorrect entities. Evaluation of five models reveals substantial variation in context utilisation behaviour, exposing a gap in existing ASR benchmarks that test only fixed prompting conditions.
Automated Benchmark Auditing for AI Agents and Large Language Models (ABA)
The paper introduces Auto Benchmark Audit (ABA), an agentic framework that systematically audits AI benchmark tasks for issues such as ambiguous specifications, environment conflicts, and incorrect ground truths. Applied to 168 benchmarks across nine domains including NeurIPS publications, ABA identifies critical issues in over 25.7% of evaluated tasks. The authors demonstrate that filtering out flawed tasks materially shifts model rankings and improves average performance on SWE-bench Verified and Terminal-Bench 2 by 9.9% and 9.6% respectively, indicating that current benchmark scores are significantly distorted by task quality problems. The agentic tool and annotations are released publicly.
Introducing SimpleQA: OpenAI's Factuality Benchmark for Language Models
OpenAI has released SimpleQA, a benchmark designed to measure language model factuality on short, fact-seeking questions. The benchmark targets a specific and well-defined capability: answering direct factual queries accurately. It is intended to provide a clean signal on model truthfulness and calibration for this class of questions.
HuggingFace and IISc Partner to Advance AI for India's Diverse Languages
Hugging Face and the Indian Institute of Science (IISc) have announced a collaboration aimed at building and improving AI models for India's many languages. The partnership focuses on expanding multilingual and low-resource language capabilities within the open-source AI ecosystem. This initiative reflects growing institutional investment in non-English language AI infrastructure, particularly for the Indian subcontinent.
Introducing HealthBench
OpenAI has released HealthBench, a new evaluation benchmark designed to assess AI model performance and safety in healthcare settings. The benchmark was developed with input from over 250 physicians and targets realistic clinical scenarios. It aims to establish a shared standard for measuring how well AI models handle health-related tasks.
OpAI-Bench: Benchmark for detecting AI text across progressive human-AI co-editing workflows
Researchers introduce OpAI-Bench, a benchmark for studying AI-text detection across progressive human-to-AI document revision workflows, covering document, sentence, token, and span granularities. Starting from human-written documents, the benchmark constructs nine sequentially revised versions per sample under five AI edit operations and varying AI coverage levels across four domains. Key findings include that mixed-authorship intermediate versions are often harder to detect than fully human or heavily AI-edited endpoints, revealing non-monotonic detection patterns absent from existing benchmarks. The work addresses a gap in AI-text detection research as real-world documents increasingly result from iterative human-AI co-editing rather than pure generation.
OpenAI introduces LifeSciBench, a life sciences AI evaluation benchmark
OpenAI has released LifeSciBench, a benchmark designed to evaluate AI systems on real-world life science research tasks and decisions. The benchmark is described as expert-authored and expert-reviewed, targeting domain-specific evaluation in biology and related fields. This addresses a gap in specialized scientific benchmarking for AI systems.
Rethinking how we measure AI intelligence
DeepMind has announced Game Arena, a new open-source evaluation platform designed for rigorous head-to-head comparison of frontier AI models. The platform uses environments with clear winning conditions to assess model capabilities. This represents DeepMind's contribution to addressing ongoing concerns about the adequacy of existing AI benchmarks.