5arXiv cs.LG (Machine Learning)·
Change2Task converts repository pull requests into verified coding agent training and evaluation tasks
Change2Task is a system that mines merged pull requests from repository history and converts them into verified, executable coding agent tasks paired with realistic software environments. The system uses three reconstruction strategies (Patch Reversal, Code Mapping, Agent Reconstruction) and achieves 79.6% verified task construction success across five task families including Bug Fix, Feature Addition, and Security Repair. On a matched candidate set it recovers 29.2% more verified tasks than a pull-request baseline, while reusing modern repository bases reduces pipeline expenditure by 10.8%. The work addresses a key bottleneck in scaling coding agents: the supply of high-quality, executable training and evaluation data.
Related guides (3)
Related events (8)
TestEvo-Bench: Live executable benchmark for test and code co-evolution tasks
Researchers introduce TestEvo-Bench, a benchmark of 1,255 tasks (746 test generation, 509 test update) mined from 152 open-source Java projects, designed to evaluate whether AI agents can correctly propagate code changes into test suites. Each task is anchored to a real commit and packaged with execution environments, enabling pass rate, coverage, and mutation score metrics. The benchmark is 'live' — new tasks are periodically mined and timestamped to allow evaluation restricted to post-training-cutoff data, reducing leakage risk. Experiments with Claude Code, Gemini CLI, and SWE-Agent paired with Claude Opus 4.7 and Gemini 3.1 Pro show up to 77.5% success on test generation, but performance drops notably on the most recent tasks and under cost constraints.
ProjAgent introduces procedural similarity retrieval for repository-level code generation
ProjAgent is a new repository-level code generation system that adds procedural similarity as an explicit retrieval signal, decomposing target functions into intermediate reasoning steps to find analogous implementations across a codebase. It combines this with conventional semantic retrieval and a static-analysis feedback loop for iterative code repair. Evaluated on the REPOCOD benchmark, ProjAgent achieves 41.14% Pass@1, outperforming existing retrieval-based baselines. The work identifies procedural similarity as a previously unexplored retrieval dimension for this class of coding agents.
Continual-learning evaluation on Terminal-Bench 2.0 tests whether agent optimizer gains compound across tasks
A new arXiv paper introduces a two-phase continual-learning evaluation framework built on Terminal-Bench 2.0 to test whether agent-optimization gains persist and compound when new tasks arrive over time. Three agent-harness optimization methods — GEPA, Meta Harness, and RELAI-VCL — are compared under identical budgets; all improve in static single-phase settings but diverge sharply under continual optimization. RELAI-VCL is the only method that both transfers positively to unseen tasks and continues improving, reaching a 76.4% lifelong average pass rate versus 58.7% for the unoptimized baseline. The key finding is that compounding gains require regression control built into the optimization loop to prevent shortcut solutions.
Probe-and-Refine Tuning improves coding agent performance via iterative repository guidance refinement
A new arXiv paper introduces probe-and-refine tuning, a procedure that uses synthetic bug-fix probes to iteratively improve AGENTS.md repository guidance files for LLM-based coding agents without requiring an agent loop during tuning. Evaluated on SWE-bench Verified with Qwen3.5-35B-A3B, the method achieves 33.0% mean resolve rate versus 28.3% for a static knowledge base baseline and 25.5% for an unguided baseline. The improvement is attributed to coverage gains—refined guidance helps agents locate the correct files rather than improving patch quality—and a step-budget experiment shows guidance is necessary for agents to productively use larger compute budgets.
SWE-Explore: New benchmark isolates repository exploration capability in coding agents
SWE-Explore is a new benchmark targeting repository exploration as a distinct, fine-grained capability of coding agents, separate from end-to-end task resolution. It covers 848 issues across 10 programming languages and 203 open-source repositories, with line-level ground truth derived from successful agent trajectories. Evaluation across retrieval methods, coding agents, and specialized localizers finds that agentic explorers outperform classical retrieval, and that line-level coverage and efficient ranking remain the key differentiators at the frontier. The benchmark addresses a gap in SWE-bench-style evaluations that treat task resolution as a binary outcome.
RuBench: Repository-level agentic coding benchmark with native Russian task specifications
RuBench 1.0 is a new benchmark of 25 repository-level agentic coding tasks drawn from real fix commits in five live open-source projects, where task specifications are written natively in Russian in the style of customer requests rather than translated from English. The benchmark evaluates deployed product configurations including Claude Code with Opus 4.8, Sonnet 5, and Haiku 4.5, and Codex CLI with GPT-5.5, with the best configuration resolving 78.7% of tasks. A notable finding is that auditing trajectories of a fifth configuration (Claude Code + Fable 5) revealed that on 20% of tasks an official safeguard fallback silently re-routed the model to Opus 4.8, providing direct evidence that the deployed product rather than the underlying model is the actual unit of measurement in agentic evaluations.
TraceLab: Real-world coding agent workload traces reveal serving optimization opportunities
Researchers at UW release TraceLab, a dataset of ~4,300 coding-agent sessions comprising ~350,000 LLM steps and ~430,000 tool calls drawn from real day-to-day use of Claude Code and Codex. Analysis reveals characteristic patterns: long autonomous loops, long contexts with short outputs, heavily-tailed tool call distributions, and high but imperfect prefix cache hit rates. The findings motivate concrete serving-system improvements including lower-overhead tool calling, append-length-aware prefill, and improved KV-cache management. The dataset, pipeline, and analysis code are publicly released.
Task-specific two-agent architecture wins QANTA 2026 multimodal QA challenge
Researchers present the top-scoring submission to the QANTA 2026 shared challenge at ICML 2026's EMM-QA Workshop, achieving an overall leaderboard score of 0.402 on multimodal quizbowl tasks. The system uses a two-agent architecture: a GPT-4.1-mini-based Tossup agent with confidence calibration and a GPT-4.1-based Bonus agent with structured relational and multimodal reasoning. Notably, the approach avoids retrieval pipelines and model ensembles, relying instead on lightweight task-specific reasoning policies under efficiency constraints. Results suggest that targeted reasoning strategies can be competitive on resource-constrained multimodal QA benchmarks.