swe-agent-b6f43761·3 events·first seen Aliases: SWE-Agent
Researchers introduce TestEvo-Bench, a benchmark of 1,255 tasks (746 test generation, 509 test update) mined from 152 open-source Java projects, designed to evaluate whether AI agents can correctly propagate code changes into test suites. Each task is anchored to a real commit and packaged with execution environments, enabling pass rate, coverage, and mutation score metrics. The benchmark is 'live' — new tasks are periodically mined and timestamped to allow evaluation restricted to post-training-cutoff data, reducing leakage risk. Experiments with Claude Code, Gemini CLI, and SWE-Agent paired with Claude Opus 4.7 and Gemini 3.1 Pro show up to 77.5% success on test generation, but performance drops notably on the most recent tasks and under cost constraints.
Three new benchmarks — DeepSWE (by Datacurve), ProgramBench (Meta/Stanford/Harvard), and ITBench-AA (IBM/Artificial Analysis) — are positioned as more rigorous replacements for the SWE-bench family, which models have largely saturated. DeepSWE tests feature implementation using private codebases and human-written problems; ProgramBench evaluates agents' ability to recreate functional programs from scratch; ITBench-AA measures root-cause diagnosis in real-world IT incident scenarios. Current top performers include GPT-5.5 (70% on DeepSWE), Claude Opus 4.7 (46.7% on ITBench-AA), and Claude Opus 4.7 (3% on ProgramBench at the 95% pass threshold), illustrating that even frontier models have substantial headroom.
Mistral AI, in collaboration with All Hands AI, releases Devstral, an agentic LLM specialized for software engineering tasks under the Apache 2.0 license. The model achieves 46.8% on SWE-Bench Verified, surpassing prior open-source state-of-the-art by over 6 percentage points and outperforming larger models like DeepSeek-V3-0324 (671B) and Qwen3 232B-A22B under the same OpenHands scaffold. Devstral is small enough to run on a single RTX 4090 or a Mac with 32GB RAM, and is available via Mistral's API at $0.1/M input tokens, as well as on HuggingFace, Ollama, and other platforms. Mistral indicates a larger agentic coding model is in development.