5arXiv cs.CL (Computation and Language)·
DeepStress: A stress-testing framework for search agent robustness to poor-quality evidence
DeepStress is a new evaluation framework that stress-tests search agents by replacing their retrieval module with a controlled synthetic environment, allowing systematic manipulation of document trustworthiness, relevance, and factuality. The authors test several search agents on HotpotQA and BrowseCompPlus, revealing substantial performance differences in handling unreliable information. The work introduces new metrics to capture system outcomes and conflicts between parametric and retrieved knowledge, addressing a gap in realistic benchmarks that rarely surface low-quality evidence scenarios.
Related guides (2)
Related events (8)
DeepWeb-Bench: A Hard Deep Research Benchmark Requiring Cross-Source Evidence and Long-Horizon Derivation
DeepWeb-Bench is a new benchmark designed to stress-test frontier language models on deep research tasks—open-web search, evidence collection, and multi-step derivation—where existing benchmarks have become saturated. The benchmark evaluates nine frontier models across four capability families (Retrieval, Derivation, Reasoning, Calibration) and finds that retrieval is not the primary bottleneck; derivation and calibration failures account for over 70% of errors. Strong models fail via incomplete derivation while weak models fail via hallucinated precision, and models show genuine domain specialization with low cross-model agreement (rho = 0.61). The benchmark, rubrics, and evaluation code are publicly released.
DeepSWE, ProgramBench, and ITBench-AA emerge as harder successors to SWE-bench for agent evaluation
Three new benchmarks — DeepSWE (by Datacurve), ProgramBench (Meta/Stanford/Harvard), and ITBench-AA (IBM/Artificial Analysis) — are positioned as more rigorous replacements for the SWE-bench family, which models have largely saturated. DeepSWE tests feature implementation using private codebases and human-written problems; ProgramBench evaluates agents' ability to recreate functional programs from scratch; ITBench-AA measures root-cause diagnosis in real-world IT incident scenarios. Current top performers include GPT-5.5 (70% on DeepSWE), Claude Opus 4.7 (46.7% on ITBench-AA), and Claude Opus 4.7 (3% on ProgramBench at the 95% pass threshold), illustrating that even frontier models have substantial headroom.
DeepRubric: Evidence-tree rubric supervision cuts RL training cost for deep research agents by 13x
DeepRubric is a data construction framework that improves reinforcement learning efficiency for deep research agents by reversing the typical rubric-generation process: rather than inferring evaluation criteria from a query, it builds an evidence tree of verifiable sub-questions first, then synthesizes aligned query-rubric pairs. The authors construct 9K training examples and train DeepRubric-8B using rubric-based GRPO, achieving comparable performance to prior open-source state-of-the-art deep research models on three benchmarks while using roughly 13x fewer RL GPU-hours. The work addresses a key bottleneck in RL-based training of long-form research agents: unreliable reward signals from incomplete rubrics.
ToolBench-X benchmarks LLM agents under tool-environment unreliability
A new arXiv preprint introduces ToolBench-X, a benchmark for evaluating LLM agents under five structured hazard types including Specification Drift, Invocation Error, Execution Failure, Output Drift, and Cross-source Conflict. Each injected hazard remains solvable via recovery paths such as retrying, fallback, or cross-checking, enabling measurement of agent resilience rather than just function-call accuracy. Experiments reveal a substantial reliability gap: agents that perform well in clean environments frequently fail under recoverable hazards, with failures driven by poor hazard diagnosis rather than insufficient tool-use volume or inference budget. The findings argue for shifting tool-use evaluation toward task completion under realistic, unreliable conditions.
Open-source DeepResearch – Freeing our search agents
Hugging Face published a blog post introducing Open Deep Research, an open-source replication of agentic deep research capabilities (similar to OpenAI's Deep Research). The project aims to build open-weight search agents capable of multi-step web research and synthesis. The post details the architecture, tooling, and early benchmark results of the system.
Data Points: DeepSWE Benchmark, DeepSeek V4 Price Cuts, MAI-Image-2.5, Mythos Security Findings, MCP Stateless Update
This edition of The Batch covers five distinct AI developments: Datacurve's DeepSWE benchmark claims to fix critical grading flaws in SWE-bench Pro with hand-written verifiers and harder tasks; DeepSeek permanently cuts V4 Pro prices by 75%; Microsoft's MAI-Image-2.5 debuts third on the Arena leaderboard; Anthropic's Claude Mythos Preview found over 10,000 high/critical vulnerabilities in the first month of Project Glasswing, with remediation badly lagging discovery; and the Model Context Protocol proposes removing stateful sessions to enable stateless, load-balanced remote servers. Each item reflects meaningful movement in evaluation methodology, inference economics, multimodal generation, AI-assisted security, and agent tooling infrastructure.
SelectBench and DAPO post-training for selective evidence adoption in RAG contexts
Researchers introduce SelectBench, a benchmark and training set for evaluating whether retrieval-augmented LLMs can selectively adopt valid evidence while rejecting misleading or injected content. They post-train Qwen3.5-4B using DAPO with rule-based and semantic judge rewards, achieving modest but directional improvements on SelectBench-v2 (22.46% to 26.46% strict success). Gains do not survive Holm multiple-comparison correction, and prompt-injection resistance shows no improvement, leaving statistical robustness and injection resistance as open challenges. General capabilities on MMLU and HotpotQA are preserved.
SearchOS: Multi-agent framework for robust open-domain information-seeking with explicit shared state
SearchOS is a new multi-agent framework that addresses a core failure mode in web-search-augmented LLM agents: repetitive search loops caused by implicit, untracked task progress. The system externalizes agent state into structured components (Frontier Task, Evidence Graph, Coverage Map, Failure Memory) via a Search-Oriented Context Management (SOCM) layer, and adds pipeline-parallel scheduling and a middleware harness to intercept tool interactions and recover from stalls. Evaluated on WideSearch and GISA benchmarks, SearchOS outperforms all single- and multi-agent baselines across all reported metrics.