5arXiv cs.CL (Computation and Language)·
Typed loop contracts for agentic code repair: evidence that stale traces degrade correctness retention
A new arXiv preprint studies the reliability gap in generate-test-revise loops used by coding agents, finding that forced revision cycles cause current correctness to drop from 0.820 to 0.673 even as ever-correct rises to 0.847. Controlled experiments with 2,430 branches show stale traces harm 34/135 correct starts versus 4/135 with current traces, a statistically significant 22.2-point increase. The authors formalize the problem by separating admission, preservation, and certification concerns, then derive a typed loop contract with a mechanically enforceable reference implementation that binds verifier evidence to exact code states. The work is framed explicitly as a specification artifact rather than a claim of improved repair competence.
Related guides (3)
Related events (8)
PoPE: Placebo-controlled evaluation of error-conditioned self-repair in frozen small code LLMs
A preregistered arXiv preprint introduces PoPE (Popperian Placebo-controlled Evaluation), a methodology for rigorously measuring whether execution error feedback actually improves code generation in frozen small LLMs (0.5–1.5B parameters). The study uses channel-specific placebos—ablating error content while preserving scaffold form—to test both prompt-based and adapter-based self-repair. Results across both channels failed to confirm content-attributable superiority over placebos or baselines, suggesting that apparent self-repair gains in prior work may reflect form rather than semantic error content. The paper argues that writing oracle-derived representations back into generation state replaces testing with conditioning.
Post-hoc falsification operators for frozen small code models fail to beat Best-of-N in leakage-free evaluation
A measurement study evaluates 26 post-hoc operators (selection, verification, repair, elimination, portfolios) applied to frozen small code models (≤1.5B parameters) against a Best-of-N baseline under a strict leakage-free, matched-compute protocol. None of the semantic operators improves held-out accuracy over BoN, with the failure traced to three structural mechanisms: a coverage wall, a capability scissors, and a near-empty consensus trap. Two non-semantic operators do provide value: an expression-layer recovery method (M1) lifts DeepSeek-Coder-1.3B by +12 tasks on HumanEval+ (p=2.4e-4), and an adaptive consensus early-stop saves ~19% compute with no accuracy harm. The paper's core lesson is that harness quality and coverage measurement should precede investment in semantic post-hoc reasoning.
Probe-and-Refine Tuning improves coding agent performance via iterative repository guidance refinement
A new arXiv paper introduces probe-and-refine tuning, a procedure that uses synthetic bug-fix probes to iteratively improve AGENTS.md repository guidance files for LLM-based coding agents without requiring an agent loop during tuning. Evaluated on SWE-bench Verified with Qwen3.5-35B-A3B, the method achieves 33.0% mean resolve rate versus 28.3% for a static knowledge base baseline and 25.5% for an unguided baseline. The improvement is attributed to coverage gains—refined guidance helps agents locate the correct files rather than improving patch quality—and a step-budget experiment shows guidance is necessary for agents to productively use larger compute budgets.
Empirical study finds 80% of AI agent-authored test patches lack meaningful verification logic
A large-scale empirical study of 86,156 test-file patches from 33,596 agent-authored GitHub PRs finds that 80.2% contain weak or no explicit oracle signals — meaning they execute code without verifying behavior. The study covers five coding agents (OpenAI Codex, GitHub Copilot, Devin, Cursor, and Claude Code) across 2,807 repositories, and introduces a syntactic taxonomy of eight oracle signal categories. Despite lower raw merge rates, regression analysis shows strong oracles significantly improve merge likelihood (OR=1.28), suggesting current quality gates based on test-file presence substantially overestimate verification strength.
Failed reasoning traces encode recoverability structure for test-time routing and post-training analysis
A new arXiv paper argues that failed reasoning traces from post-trained LLMs contain exploitable signal about whether failures are recoverable via resampling or require structural intervention. The authors derive three trajectory features from the distributional signature of failed rollouts (not their text content) that cluster failures into stable regimes and characterize failure topography across post-training methods with 84.3% accuracy. A training-free routing rule built on these features lifts rescue rates by +12.2% on a deployment-relevant hard subset, and the features transfer across model families. The work reframes failed traces as diagnostic objects rather than discarded data, with implications for inference-time compute allocation and post-training analysis.
TraceLab: Real-world coding agent workload traces reveal serving optimization opportunities
Researchers at UW release TraceLab, a dataset of ~4,300 coding-agent sessions comprising ~350,000 LLM steps and ~430,000 tool calls drawn from real day-to-day use of Claude Code and Codex. Analysis reveals characteristic patterns: long autonomous loops, long contexts with short outputs, heavily-tailed tool call distributions, and high but imperfect prefix cache hit rates. The findings motivate concrete serving-system improvements including lower-overhead tool calling, append-length-aware prefill, and improved KV-cache management. The dataset, pipeline, and analysis code are publicly released.
GRACE: Graph-Regularized Agentic Context Evolution for reliable long-horizon instruction updates
Researchers introduce GRACE, a method that maintains a deployed LLM agent's persistent system-level instructions as a typed semantic graph rather than flat text, enabling local verification of updates within typed node neighborhoods. Evaluated on a telecom agent harness derived from τ²-bench under distribution shift, GRACE improves pass³ reliability from 0.091 (Gemini 2.5 Flash zero-shot) to 0.673±0.136, surpassing a Gemini 3.1 Pro zero-shot reference of 0.242. The work identifies structural substrate and consolidation mechanisms as key requirements for reliable long-horizon agentic context evolution. The flat-text baseline finishes at 0.191, underscoring the practical gap GRACE addresses.
Audit finds GSO, SWE-Perf, and SWE-fficiency benchmarks unreliable for measuring coding agent progress
A new arXiv paper audits three prominent repository-level code-optimization benchmarks (GSO, SWE-Perf, SWE-fficiency) used to rank coding agents, finding significant reliability problems across all three. Reference patches satisfy validity rules in cross-machine replays for only 39/102 GSO tasks and 11/140 SWE-Perf tasks, and leaderboard rankings disagree on 9 of 28 pairwise comparisons depending on scoring rule choice. The authors also find that at least one public submission already matches or beats the reference patch on 85.3% of replay-valid tasks, suggesting aggregate leaderboard scores obscure the true frontier. The study raises substantive concerns about whether these benchmarks are providing reliable signal for claims of coding-agent capability progress.