7arXiv cs.CL (Computation and Language)·
EdgeBench finds log-sigmoid scaling laws for agent learning from real-world environments
A new arXiv preprint introduces EdgeBench, a suite of 134 real-world tasks with ultra-long horizons (12+ hours of continuous agent operation each) spanning scientific discovery, software engineering, formal mathematics, and other domains. Analyzing ~38,000 hours of agent-environment interaction, the authors report the first evidence that agent performance during environment learning follows a log-sigmoid scaling law with R²=0.998. They also find that agent learning speed roughly doubles every three months across model generations, drawing an analogy to pretraining scaling laws but for post-deployment environment learning. 51 tasks and the evaluation framework are publicly released.
Related guides (3)
Related events (8)
EurekAgent: Environment Engineering as the Key Bottleneck for Autonomous Scientific Discovery
EurekAgent is a new LLM-based agent system that reframes autonomous scientific discovery around 'environment engineering' — designing the resources, constraints, and interfaces that shape agent behavior — rather than prescribing agent workflows. The system engineers four dimensions: permissions, artifact management (filesystem/Git), budget awareness, and human-in-the-loop oversight. It achieves state-of-the-art results on mathematics, kernel engineering, and ML tasks, including new 26-circle packing results at under $11 in API cost, and is fully open-sourced.
EnvFactory: Scaling Tool-Use Agents via Executable Environments Synthesis and Robust RL
EnvFactory is a fully automated framework for training tool-use LLM agents via Agentic Reinforcement Learning, addressing two key bottlenecks: scalable execution environments and realistic multi-turn training data. It autonomously constructs stateful, executable tool environments from authentic resources and synthesizes natural trajectories with implicit human intents via topology-aware sampling. Using only 85 verified environments across 7 domains, it generates 2,575 SFT and RL trajectories and improves Qwen3-series models by up to +15% on BFCLv3, +8.6% on MCP-Atlas, and +6% on conversational benchmarks, outperforming prior approaches that use 5x more environments.
EvoPolicyGym benchmark evaluates autonomous policy evolution by LLM agents in RL environments
A new arXiv preprint introduces EvoPolicyGym, a benchmark for evaluating how LLM-based agents iteratively improve executable policies in compact interactive RL environments under a fixed interaction budget. The benchmark provides trajectory-level diagnostics beyond aggregate scores, distinguishing how agents allocate budget and convert feedback into parametric tuning. GPT-5.5 achieves the strongest aggregate rank score and top-two performance across all 16 environments. The work targets a gap in agent evaluation where iterative policy refinement is conflated with open-ended software engineering progress.
Agents-A1: 35B MoE agent matches trillion-parameter models via horizon scaling
Researchers introduce Agents-A1, a 35B Mixture-of-Experts model that claims to match or exceed trillion-parameter models like Kimi-K2 and DeepSeek V4 on long-horizon agentic benchmarks. The approach scales agent trajectory length (averaging 45K tokens) and heterogeneous agent abilities rather than raw parameter count, using a three-stage training recipe including multi-teacher domain-routed distillation. On benchmarks such as SEAL-0, IFBench, HiPhO, and FrontierScience-Olympiad, Agents-A1 achieves leading or competitive results against models with roughly 30x more parameters. The work proposes a practical efficiency path for agentic capability scaling without proportional compute scaling.
NatureBench: Coding agents surpass published SOTA on only 17.8% of real scientific tasks from Nature-family papers
NatureBench introduces a 90-task benchmark derived from peer-reviewed Nature-family publications to evaluate whether AI coding agents can advance beyond reproduction toward genuine scientific discovery. Built on NatureGym, an automated pipeline that creates containerized per-task environments, the benchmark addresses environment fragmentation that has undermined prior agent-on-research evaluations. Evaluating ten frontier agent configurations under a web-search-disabled protocol, the strongest model exceeds published SOTA on only 17.8% of tasks, with failures driven primarily by wrong method choice and insufficient compute rather than task misunderstanding. Agents succeed mainly through methodological translation—recasting scientific problems as supervised prediction—rather than genuine scientific invention.
Agentopia: Long-term multi-agent life simulation framework for training LLMs on social behavior
Researchers introduce Agentopia, a framework for simulating 10 years of social life across 100 LLM-powered agents, enabling study of emergent social behaviors and long-term personal growth dynamics. The system defines a 'life reward' metric mirroring human well-being and uses it to train LLMs via rejection sampling. Training on simulated social experience yields a +15.6% improvement on downstream role-playing benchmarks, suggesting that synthetic social simulation can generalize to real capability gains.
GLM-5.1 Open-Weights Model Targets Long-Running Agentic Tasks; Andrew Ng on Coding Agent Acceleration by Software Domain
Z.ai released GLM-5.1, an open-weights mixture-of-experts LLM (754B total / 40B active parameters) designed for sustained agentic coding tasks lasting up to eight hours, featuring iterative planning-execution-evaluation loops with thousands of tool calls. The model claims top open-weights performance on Artificial Analysis Intelligence Index and SWE-Bench Pro, available under MIT license via HuggingFace. The accompanying editorial by Andrew Ng offers a tiered framework for how much coding agents accelerate different software work categories—frontend most, then backend, infrastructure, and research least—with practical implications for team organization. A secondary item references data-center opposition and LLM helpfulness failure modes.
PlanBench-XL: Benchmark for LLM Agent Planning in Large-Scale Tool Ecosystems
Researchers introduce PlanBench-XL, an interactive benchmark of 327 retail tasks spanning 1,665 tools designed to evaluate LLM agents on long-horizon planning under retrieval-limited tool visibility. The benchmark includes a blocking mechanism simulating real-world disruptions such as missing or failing tools, forcing agents to detect and recover from broken execution paths. Experiments on ten leading LLMs reveal severe performance degradation: GPT-5.4 drops from 51.90% accuracy in unblocked settings to 11.36% under the most severe blocking condition, highlighting fragility in adaptive planning for large, imperfect tool environments.