5arXiv cs.AI (Artificial Intelligence)·
IdeaGene-Bench: Benchmark for Scientific Lineage Reasoning and Idea Generation in LLMs
Researchers introduce IG-Bench, a benchmark evaluating whether AI systems can trace and reason about the evolutionary inheritance structure of scientific ideas across papers. The benchmark contains 1,961 lineage traces, 1,085 Idea Genome objects, and 920 pairwise GenomeDiff records spanning 10 scientific domains, supporting both closed-form reasoning tasks and lineage-conditioned generation evaluation. Experiments on 14 LLM-based systems reveal a compositional bottleneck: the best system achieves only 27.3% exact accuracy on lineage reasoning, and adding structured lineage context reshuffles system rankings rather than uniformly improving performance.
Related guides (1)
Related events (8)
SkillGenBench: Benchmarking Skill Generation Pipelines for LLM Agents
SkillGenBench is a new benchmark designed to evaluate the ability of LLM agents to generate correct, reusable, and executable skills from raw repositories and documents, rather than merely using pre-provided skills. It covers two generation regimes (task-conditioned and task-agnostic) and two procedural sources (repository-grounded and document-grounded), with standardized execution-based evaluation protocols. Experiments across multiple skill-generation methods reveal substantial performance variation and distinct failure modes depending on source type. The benchmark aims to establish skill generation as an independent research problem within agent systems.
OpenAI introduces GeneBench-Pro, a genomics and biology AI benchmark
OpenAI has released GeneBench-Pro, a new benchmark designed to evaluate AI performance on genomics, biology, and scientific research tasks using complex, real-world datasets. The announcement comes from OpenAI's official blog, positioning it as a domain-specific evaluation tool for scientific AI. This extends the evaluation landscape into life sciences, a growing area of AI application.
AGC-Bench: A unified benchmark for measuring artificial general creativity in LLMs
Researchers introduce AGC-Bench, a comprehensive AI creativity benchmark built from a systematic review of 3,101 papers and 497 existing benchmarks, covering 78 datasets across brainstorming, STEM, narrative, figurative language, and humor. The work introduces Judge Response Theory to correct for LLM-as-judge bias and fine-tunes Qwen3-30B to produce AGC-Judge, an open-weight scoring model. Key findings include the recovery of a single creativity factor 'c' (analogous to the general intelligence 'g' factor) explaining 81.5% of variance across 83 LLMs, and evidence that top humans still outperform top LLMs on creativity tasks. The benchmark, leaderboard, and human data are released as open infrastructure.
OmniaBench: A broad diagnostic benchmark for general AI agents across 354 domains
OmniaBench is a new benchmark for evaluating general AI agents across diverse scenarios, spanning 90 level-1 and 354 level-2 domains derived from app stores, product documents, and web retrieval. The benchmark contains 1,431 tasks with single-turn and multi-turn formats, a ten-dimensional capability taxonomy, and eight atomic difficulty factors for fine-grained analysis. Even frontier models like Claude Sonnet 5 and GPT-5.6-Sol achieve only ~58% Overall Pass@1, revealing persistent weaknesses in planning, constraint maintenance, and adaptive correction. The work addresses a gap in existing agent benchmarks that tend to cover narrow tool ecosystems or interaction formats.
SoundnessBench: Benchmarking LLMs as Evaluators of ML Research Proposal Viability
SoundnessBench is a new benchmark of 1,099 machine-learning research proposals derived from ICLR submissions, labeled with reviewer soundness scores, designed to test whether LLMs can reliably distinguish methodologically sound research ideas from unsound ones. Evaluated across 12 frontier LLMs, the benchmark reveals a pervasive optimism bias: models systematically rate low-soundness proposals as sound under standard prompting, with aggressive prompting shifting errors from false positives to false negatives rather than eliminating them. Controls for data contamination, surface features, and human audit quality suggest the bias is not attributable to a single confounder. The authors conclude that current LLMs are not yet reliable as standalone first-gate evaluators of scientific rigor, a critical bottleneck for autonomous AI research agents.
NatureBench: Coding agents surpass published SOTA on only 17.8% of real scientific tasks from Nature-family papers
NatureBench introduces a 90-task benchmark derived from peer-reviewed Nature-family publications to evaluate whether AI coding agents can advance beyond reproduction toward genuine scientific discovery. Built on NatureGym, an automated pipeline that creates containerized per-task environments, the benchmark addresses environment fragmentation that has undermined prior agent-on-research evaluations. Evaluating ten frontier agent configurations under a web-search-disabled protocol, the strongest model exceeds published SOTA on only 17.8% of tasks, with failures driven primarily by wrong method choice and insufficient compute rather than task misunderstanding. Agents succeed mainly through methodological translation—recasting scientific problems as supervised prediction—rather than genuine scientific invention.
AdversaBench: Automated LLM red-teaming pipeline with multi-judge confirmation and cross-model transferability
AdversaBench is a new end-to-end red-teaming pipeline that mutates seed prompts using five structured operators and confirms failures via a three-judge panel with a meta-judge tiebreaker. Experiments on 45 seeds across reasoning, instruction-following, and tool-use categories produced confirmed failures for every seed. Key findings include sharp variation in operator effectiveness by category, misleading binary failure rates, judge agreement metrics distorted by label skew, and zero-shot transferability of adversarial prompts from Llama 3.1 8B to Llama 3.3 70B. Code and dataset are publicly released.
AI-Assisted Systematization for Evaluating GenAI Systems
This paper addresses a foundational gap in GenAI evaluation: the underspecification of broad, contested concepts like 'reasoning,' 'fairness,' or 'creativity.' The authors introduce a structured artifact called a 'concept spec' and a validation worksheet, then build two AI-assisted systematizers—a zero-shot approach and a multi-agent approach—to convert vague evaluation targets into measurable, structured accounts. They apply these tools to hate-based rhetoric and digital empathy, assessing the resulting specs on content validity and information recoverability. The work positions AI assistance as a scalable aid for the cognitively demanding process of evaluation design.