7arXiv cs.AI (Artificial Intelligence)·
Google's Paper Assistant Tool (PAT) automates scientific peer review with agentic AI
Google introduces the Paper Assistant Tool (PAT), an agentic AI framework that ingests full scientific manuscripts and produces comprehensive evaluations including theoretical checking, experiment validation, and flaw identification. PAT uses inference scaling techniques to achieve a 34% improvement over zero-shot recall on mathematical errors in the SPOT benchmark. The system was piloted as a pre-submission tool at two major CS conferences (STOC and ICML), demonstrating practical deployment at scale. The paper also proposes a four-level taxonomy of AI-human collaboration in scientific evaluation.
Related guides (4)
Related events (8)
PaperBench: OpenAI Benchmark for Evaluating AI Agents on Research Replication
OpenAI introduces PaperBench, a benchmark designed to evaluate AI agents' ability to replicate state-of-the-art AI research papers end-to-end. The benchmark targets a high-complexity capability: reproducing experimental results from frontier AI research, which requires code generation, experimental design, and scientific reasoning. This positions PaperBench as a tool for tracking progress toward autonomous AI research agents.
Automated Benchmark Auditing for AI Agents and Large Language Models (ABA)
The paper introduces Auto Benchmark Audit (ABA), an agentic framework that systematically audits AI benchmark tasks for issues such as ambiguous specifications, environment conflicts, and incorrect ground truths. Applied to 168 benchmarks across nine domains including NeurIPS publications, ABA identifies critical issues in over 25.7% of evaluated tasks. The authors demonstrate that filtering out flawed tasks materially shifts model rankings and improves average performance on SWE-bench Verified and Terminal-Bench 2 by 9.9% and 9.6% respectively, indicating that current benchmark scores are significantly distorted by task quality problems. The agentic tool and annotations are released publicly.
AI-Assisted Systematization for Evaluating GenAI Systems
This paper addresses a foundational gap in GenAI evaluation: the underspecification of broad, contested concepts like 'reasoning,' 'fairness,' or 'creativity.' The authors introduce a structured artifact called a 'concept spec' and a validation worksheet, then build two AI-assisted systematizers—a zero-shot approach and a multi-agent approach—to convert vague evaluation targets into measurable, structured accounts. They apply these tools to hate-based rhetoric and digital empathy, assessing the resulting specs on content validity and information recoverability. The work positions AI assistance as a scalable aid for the cognitively demanding process of evaluation design.
AutoResearchClaw: Fully Autonomous Self-Evolving Research Agent (Idea to Paper)
AutoResearchClaw is an open-source Python project from aiming-lab that claims to automate the full research pipeline from idea to paper, positioning itself as a fully autonomous and self-evolving research agent. The repository has accumulated 12,426 stars with 55 added today, indicating notable community traction. It represents a concrete implementation in the growing space of AI agents designed to conduct and write scientific research autonomously.
Measuring AI's capability to accelerate biological research
OpenAI introduces a real-world evaluation framework designed to measure how AI systems can accelerate biological research in wet lab settings. The work uses GPT-5 to optimize a molecular cloning protocol as a concrete demonstration case. The framework explicitly addresses both the potential benefits and biosecurity risks of AI-assisted experimentation, positioning this as a dual-use capability assessment.
AiraXiv: AI-Driven Open-Access Publishing Platform for Human and AI Scientists
AiraXiv is a proposed open-access academic publishing platform designed to accommodate both human and AI-generated research outputs, addressing scalability challenges in traditional peer review. The platform supports AI scientists via Model Context Protocol (MCP)-based interactions and human scientists through an interactive UI, with papers evolving through continuous feedback-driven iteration. It was validated through real-world deployment as the submission platform for ICAIS 2025. The work positions itself as infrastructure for a future where AI agents are first-class participants in the scientific publishing ecosystem.
Fine-tuned PEGASUS-large outperforms LLaMA-3 and GPT-3.5 for automatic research paper title generation
Researchers propose a system for generating research paper titles from abstracts using pre-trained and large language models, evaluated on CSPubSum, LREC-COLING-2024, and a new dataset SpringerSSAT. Fine-tuned PEGASUS-large outperforms fine-tuned LLaMA-3-8B and zero-shot GPT-3.5-turbo across most metrics including ROUGE, METEOR, BERTScore, and SciBERTScore. The work is a narrow NLP application study with limited broader implications for the AI/ML landscape.
Consensus accelerates research with GPT-5 and Responses API
Consensus, an AI-powered academic research platform with over 8 million users, has integrated GPT-5 and OpenAI's Responses API to build a multi-agent research assistant. The system reads, analyzes, and synthesizes scientific evidence in minutes. This represents a production deployment of GPT-5 in a domain-specific, agentic research workflow.