6arXiv cs.CL (Computation and Language)·
Systematic evaluation finds no universally superior autonomous discovery harness; adaptive allocation outperforms fixed choices
A new arXiv preprint systematically decomposes OpenEvolve-style evolutionary search and TTT-Discover into constituent components, evaluating 30 budget-matched harnesses across 12 model-problem pairs using over 3.1 million LLM rollouts. Results show no fixed harness generalizes reliably across settings, and simpler alternatives often outperform OpenEvolve variants. The authors demonstrate that early discovery progress predicts final performance and use this to motivate an adaptive budget-allocation strategy that prunes weak runs and reallocates compute to stronger survivors, outperforming fixed harness selection.
Related guides (2)
Related events (8)
Self-evolving agent harness framework uses quality-diversity archive to generalize harness improvements across model families
A new arXiv preprint introduces a framework for automatically improving LLM agent harnesses (prompts, injected knowledge, runtime control, configuration) without modifying model weights. The system separates change proposal (done by an LLM diagnosing failures) from credit assignment (done by deterministic statistical testing), populating a gated quality-diversity archive keyed on failure pathology rather than task identity to prevent overfitting. Evaluated across seven domains with a frozen open-weight model, the framework achieves +9 to +15.5 percentage point gains on sealed test sets, retaining 86–147% of training gains, with the key finding that pathology-to-patch mappings transfer across model families.
Continual-learning evaluation on Terminal-Bench 2.0 tests whether agent optimizer gains compound across tasks
A new arXiv paper introduces a two-phase continual-learning evaluation framework built on Terminal-Bench 2.0 to test whether agent-optimization gains persist and compound when new tasks arrive over time. Three agent-harness optimization methods — GEPA, Meta Harness, and RELAI-VCL — are compared under identical budgets; all improve in static single-phase settings but diverge sharply under continual optimization. RELAI-VCL is the only method that both transfers positively to unseen tasks and continues improving, reaching a 76.4% lifelong average pass rate versus 58.7% for the unoptimized baseline. The key finding is that compounding gains require regression control built into the optimization loop to prevent shortcut solutions.
Claw-SWE-Bench: A benchmark for evaluating agent harnesses on multilingual coding tasks
Researchers introduce Claw-SWE-Bench, a multilingual SWE-bench-style benchmark and adapter protocol designed to fairly compare heterogeneous agent harnesses ("claws") on GitHub issue-resolution tasks. The benchmark contains 350 instances across 8 languages and 43 repositories, with an 80-instance Lite subset for cost-efficient validation. Key findings show adapter design dominates raw model choice: a minimal adapter scores 19.1% Pass@1 versus 73.4% for a full adapter using the same GLM 5.1 backbone, and harness choice and model choice each shift Pass@1 by roughly 27-29 percentage points. The work also introduces cost accounting as a first-class evaluation axis alongside accuracy.
Evolution Strategies as a Scalable Alternative to Reinforcement Learning
OpenAI published research showing that evolution strategies (ES), a decades-old optimization technique, can match standard reinforcement learning performance on benchmarks like Atari and MuJoCo. The approach offers practical advantages over RL including easier parallelization and fewer hyperparameter sensitivities. This positions ES as a viable alternative training paradigm for policy optimization tasks.
From Model Scaling to System Scaling: Scaling the Harness in Agentic AI
This paper argues that the next major bottleneck in agentic AI is system-level design—what the authors call 'scaling the harness'—rather than continued model scaling alone. The agent harness encompasses memory substrates, context constructors, skill-routing layers, orchestration loops, and verification/governance components that together translate model capability into long-horizon behavior. The authors identify three core bottlenecks (context governance, trustworthy memory, dynamic skill routing) and propose harness-level benchmarks measuring trajectory quality, memory hygiene, and verification cost. They introduce CheetahClaws, a Python-native reference harness, and compare it against Claude Code and OpenClaw.
EvoPolicyGym benchmark evaluates autonomous policy evolution by LLM agents in RL environments
A new arXiv preprint introduces EvoPolicyGym, a benchmark for evaluating how LLM-based agents iteratively improve executable policies in compact interactive RL environments under a fixed interaction budget. The benchmark provides trajectory-level diagnostics beyond aggregate scores, distinguishing how agents allocate budget and convert feedback into parametric tuning. GPT-5.5 achieves the strongest aggregate rank score and top-two performance across all 16 environments. The work targets a gap in agent evaluation where iterative policy refinement is conflated with open-ended software engineering progress.
Systematic framework for selecting trajectories in data augmentation evaluated across five strategies
A thesis-derived arXiv preprint proposes a framework for evaluating five trajectory selection strategies—Outlierness, Diversity, Representativeness, Uncertainty, and Random—for data augmentation in spatio-temporal ML tasks. The study tests these strategies across four datasets spanning animal behavior, maritime, and urban traffic domains using linear and non-linear models with Optuna-based hyperparameter optimization. Key findings show systematic strategies (especially Outlierness and Uncertainty) outperform random selection in sparse datasets but can degrade performance in dense, high-quality datasets, with UMAP visualization confirming topological effects.
MLEvolve: Self-evolving multi-agent framework for automated ML algorithm discovery
MLEvolve is a new LLM-based multi-agent framework for end-to-end machine learning algorithm discovery, addressing limitations of existing MLE agents including information isolation and memoryless search. The system introduces Progressive MCGS (a graph-extended tree search), Retrospective Memory for experience accumulation, and decoupled strategic planning from code generation. Evaluated on MLE-Bench, it achieves state-of-the-art medal and valid submission rates within a 12-hour budget, and also outperforms AlphaEvolve on mathematical algorithm optimization tasks.