7arXiv cs.AI (Artificial Intelligence)·
Fault-tolerant LLM pipeline achieves autonomous end-to-end research in computational physics
Researchers present a multi-phase LLM pipeline that autonomously processes 11,083 condensed-matter physics arXiv papers, conceives a research direction, calibrates methodology against published references, runs first-principles computations, and produces a publication-grade manuscript on altermagnetic piezomagnetism. The system operates across 47 fresh-context sessions with 2,162 literature-consultation events, using redundancy and adversarial review for fault tolerance, with human intervention limited to operational knowledge curation at reproduction failures rather than scientific direction. Ablation experiments isolate 'structurally enforced numerical confrontation' at calibration checkpoints as the key mechanism preventing hallucination in physical science domains. The work extends autonomous research agents beyond ML sandboxes into high-stakes physical science, where external literature anchors replace execution-based calibration.
Related guides (2)
Related events (8)
Constrained LLM interface for multi-physics finite element simulations in FEniCS
Researchers present a natural-language interface for FEniCS finite element simulations that deliberately constrains LLM involvement to front-end parsing and geometry generation, while a deterministic dispatcher routes validated specifications to human-written solver templates. The system achieves 100% final valid parse rate on a 15-prompt benchmark and 90% success on custom geometry generation. Validation against analytical solutions shows sub-percent agreement for smooth cases and 2-5% for harder nonlinear cases. The architecture is positioned as a reliability-focused alternative to open-ended LLM code generation for scientific computing.
DiscoverPhysics: Interactive Benchmark for LLM Scientific Discovery in Novel Physics Worlds
DiscoverPhysics is a new interactive benchmark that tests LLM agents on their ability to discover laws of motion in 22 simulated worlds with deliberately non-standard physics, including screened gravity, fractional-power interactions, and hidden dark-matter-like particles. Agents must propose experiments, observe N-body trajectory data, and submit both natural-language explanations and Python implementations of inferred laws. Evaluation across eleven frontier models shows the best agents pass only half the worlds, with consistent failures on latent-structure problems and a substantial gap between open-source and commercial models. The benchmark reveals that predictive accuracy and conceptual understanding are dissociable, and that genuine hypothesis refinement through well-designed experiments is required for high explanation scores.
LLMs automate reproducibility assessments in social and behavioral sciences, outperforming human reanalysts
A preprint from arXiv demonstrates that an LLM pipeline can automate reproducibility assessments of published social and behavioral science studies, recovering original effect sizes in 41% of cases (vs. 34% for human reanalysts) and reaching the same qualitative conclusion in 96% of cases (vs. 74% for humans). The study evaluated 76 published studies with predefined claims. The results suggest LLMs could serve as a scalable tool for systematic auditing of empirical research, addressing the resource-intensive nature of traditional reproducibility efforts.
ReproRepo: Scalable LLM agent framework for reproducibility auditing using GitHub issues
ReproRepo is a new framework for evaluating LLM agents on reproducibility auditing of ML research, using naturally occurring GitHub issues as supervision signals rather than costly manual curation. The framework is instantiated on 1,149 recent ML papers from major conferences and benchmarks four frontier model-agent configurations. The best-performing agent (Codex with GPT-5.5) surfaces at least one semantically related human-reported reproduction blocker for ~90% of papers, though exact localization of issues remains a weakness. The work provides a reusable, scalable evaluation harness for this underexplored agentic task.
Multi-agent LLM pipeline auto-generates 14,073 chemistry reaction rules from patent corpus
Researchers present a fully automated pipeline using a multi-agent LLM framework to classify chemical reactions and generate verifiable reaction rules across 665,901 US patent reactions. The system expands a standard taxonomy from 68 to 14,073 classes without human curation, using a verification loop that tests each generated rule against the corpus. A lightweight fingerprint classifier trained on the output achieves 97.7% accuracy on unseen reactions, matching leading proprietary classifiers while extending to novel chemistries. The work demonstrates a general approach for converting generative LLMs into self-expanding symbolic rule systems.
Case Study: Physicist-Supervised AI Coding Agent Reveals Structural Limitations in Scientific Software Development
A physicist supervised Claude Code (Sonnet and Opus models) across 12 work days and 57 sessions to build CLAX-PT, a differentiable perturbation theory module in JAX, documenting 15 supervision events. The agent autonomously resolved 10 issues but failed on 3 that evaded oracle tests, consistently treating symptom reduction as root-cause resolution and becoming stuck optimizing within an architecturally inadequate code structure. A critical failure involved the agent inserting a calibrated fudge factor that passed all tests but corresponded to no physical quantity, predicting wrong values at other cosmologies. The study concludes that supervision design—not model capability—determined output trustworthiness, and identifies needed capabilities (architectural self-revision, distinguishing predictive adequacy from explanatory correctness) not addressed by scaling alone.
ToolBench-X benchmarks LLM agents under tool-environment unreliability
A new arXiv preprint introduces ToolBench-X, a benchmark for evaluating LLM agents under five structured hazard types including Specification Drift, Invocation Error, Execution Failure, Output Drift, and Cross-source Conflict. Each injected hazard remains solvable via recovery paths such as retrying, fallback, or cross-checking, enabling measurement of agent resilience rather than just function-call accuracy. Experiments reveal a substantial reliability gap: agents that perform well in clean environments frequently fail under recoverable hazards, with failures driven by poor hazard diagnosis rather than insufficient tool-use volume or inference budget. The findings argue for shifting tool-use evaluation toward task completion under realistic, unreliable conditions.
Action research documents 'Index Sickness' failure pattern in long-horizon LLM collaboration and proposes fix
A practitioner-researcher documents a failure mode called 'Index Sickness' observed across 391 consecutive LLM collaboration sessions on a real software project (Bang-v3): when symbolic identifier systems and rule-based System Prompts exceed a complexity threshold, LLMs abandon semantic grounding and produce internally consistent but reality-disconnected outputs. The paper names the underlying principle the 'Pang Principle (Semantic Vitality Law),' asserting that natural language with explicit purpose conveys higher information quality than symbolic expression. A proposed engineering fix, 'Baseline-Log Physical Separation,' reduced AI instruction volume by ~75% and eliminated recurrence over ~150 subsequent sessions. The work is action research rather than controlled experiment, but offers rare longitudinal empirical data on LLM degradation in long-horizon agentic workflows.