AI's Capability in Assisting Scientific Research in Physics, Astrophysics, and Cosmology II: Project Planning and Proposal Evaluation

Study finds LLMs produce competitive physics research proposals but AI reviewers show systematic bias toward AI-written content

A new arXiv preprint evaluates how well LLMs (ChatGPT, Claude, DeepSeek) can generate one-page research project plans in physics, astrophysics, and cosmology, comparing them against human-written proposals across 32 total documents. Human reviewers rated human and AI proposals similarly and correctly identified authorship ~72-79% of the time, while AI reviewers (Claude Opus 4.8 and ChatGPT Pro 5.5) correctly classified all 32 proposals and scored AI-written proposals roughly one point higher than human-written ones on a five-point scale. The study raises a concrete concern about deploying LLMs in grant proposal evaluation pipelines, as AI reviewers exhibit a systematic preference for AI-generated content.