5arXiv cs.CL (Computation and Language)·
Spider 2.0-AIFunc benchmark evaluates text-to-SQL for AI-native SQL workflows on Snowflake
Researchers introduce Spider 2.0-AIFunc, a benchmark of 465 verified instances across 125 real-world databases targeting AI-native SQL functions (classification, sentiment analysis, similarity search, etc.) natively exposed by cloud platforms like Snowflake. Existing text-to-SQL benchmarks cover only conventional SQL, leaving a gap in evaluating whether models can generate queries that invoke LLM capabilities inline. Evaluation of ten state-of-the-art models finds top proprietary models reach 67-70% execution accuracy versus 58.1% for the best open-source model, with failures concentrated in predicate specification, schema grounding, and AI function parameterization. Notably, elaborate agent frameworks designed for traditional text-to-SQL do not transfer well to this setting, with minimal agent setups matching or outperforming more complex alternatives.
Related guides (2)
Related events (8)
TAHOE: Error-driven hint learning system substantially improves Text-to-SQL on Spider 2.0
TAHOE is a Text-to-SQL system that treats prompt optimization as a dynamic data management problem, building a structured Hint Bank from compiler, execution, and user feedback without updating model parameters. On the Spider 2.0-Snow benchmark using GPT-5.5, it raises pass rate from 61.95% to 79.42% and achieves 100% Snowflake syntax compliance while reducing compiler-feedback rounds from 2.79 to 0.12. The learned Hint Bank transfers to weaker models, yielding a 19.7 percentage-point gain on Doubao-2.0-lite. The approach targets the production deployment gap between Text-to-SQL prototypes and real-world database environments with strict dialects and large schemas.
Snowflake and Anthropic Announce $200M Multi-Year Partnership for Agentic AI in Enterprise
Anthropic and Snowflake have expanded their strategic partnership into a multi-year, $200 million agreement to deploy Claude models and AI agents across Snowflake's 12,600+ global enterprise customers via Amazon Bedrock, Google Cloud Vertex AI, and Microsoft Azure. The deal centers on agentic AI capabilities including Snowflake Intelligence (powered by Claude Sonnet 4.5), Cortex AI Functions supporting multimodal queries, and Cortex Agents for multi-step data reasoning, with claimed >90% accuracy on complex text-to-SQL tasks. Snowflake customers already process trillions of Claude tokens per month through Cortex AI, and the partnership targets regulated industries including financial services, healthcare, and life sciences. Claude Code is also deployed internally across Snowflake's engineering organization.
IBM Research introduces ScarfBench for evaluating AI agents on enterprise Java framework migration
IBM Research published ScarfBench, a benchmark designed to evaluate AI agents on the task of migrating enterprise Java frameworks. The benchmark targets a concrete, high-value software engineering task relevant to large-scale enterprise codebases. It appears on the Hugging Face blog, suggesting an open release or community-facing artifact.
AARRI-Bench evaluates frontier LLMs and agents on granular research-intern-level tasks
Researchers introduce AARR (Act As a Real Researcher), a new benchmark series targeting whether AI agents can emulate the professionalism, thoroughness, and nuanced judgment of human researchers in granular research scenarios—not just macro-level task execution. The first benchmark, AARRI-Bench, tests frontier models and agentic harnesses, finding that even the best configuration (Mini-SWE-Agent with Claude Opus 4.7) achieves only 68.3% success, frequently missing subtle but critical details obvious to human researchers. The work argues that closing the gap requires deeper modeling of research behavior rather than more complex scaffolding.
ITBench-AA: Frontier Models Score Below 50% on the First Benchmark for Agentic Enterprise IT Tasks
IBM Research and Artificial Analysis have released ITBench-AA, a benchmark targeting agentic AI performance on enterprise IT operations tasks. Frontier models evaluated on the benchmark score below 50%, indicating significant capability gaps in real-world IT automation scenarios. The benchmark appears to be the first of its kind focused specifically on agentic enterprise IT workflows, covering tasks relevant to site reliability engineering and IT operations.
Data Points: DeepSWE Benchmark, DeepSeek V4 Price Cuts, MAI-Image-2.5, Mythos Security Findings, MCP Stateless Update
This edition of The Batch covers five distinct AI developments: Datacurve's DeepSWE benchmark claims to fix critical grading flaws in SWE-bench Pro with hand-written verifiers and harder tasks; DeepSeek permanently cuts V4 Pro prices by 75%; Microsoft's MAI-Image-2.5 debuts third on the Arena leaderboard; Anthropic's Claude Mythos Preview found over 10,000 high/critical vulnerabilities in the first month of Project Glasswing, with remediation badly lagging discovery; and the Model Context Protocol proposes removing stateful sessions to enable stateless, load-balanced remote servers. Each item reflects meaningful movement in evaluation methodology, inference economics, multimodal generation, AI-assisted security, and agent tooling infrastructure.
DeepSWE, ProgramBench, and ITBench-AA emerge as harder successors to SWE-bench for agent evaluation
Three new benchmarks — DeepSWE (by Datacurve), ProgramBench (Meta/Stanford/Harvard), and ITBench-AA (IBM/Artificial Analysis) — are positioned as more rigorous replacements for the SWE-bench family, which models have largely saturated. DeepSWE tests feature implementation using private codebases and human-written problems; ProgramBench evaluates agents' ability to recreate functional programs from scratch; ITBench-AA measures root-cause diagnosis in real-world IT incident scenarios. Current top performers include GPT-5.5 (70% on DeepSWE), Claude Opus 4.7 (46.7% on ITBench-AA), and Claude Opus 4.7 (3% on ProgramBench at the 95% pass threshold), illustrating that even frontier models have substantial headroom.
WrenAI: open-source text-to-SQL GenBI platform for AI agents across 20+ data sources
WrenAI is an open-source Generative BI (GenBI) tool that converts natural-language questions into SQL queries, dashboards, and charts across 20+ data sources including BigQuery, Snowflake, PostgreSQL, and Databricks. It positions itself as a governed text-to-SQL layer for AI agents via an open context layer. The project has accumulated 16,101 GitHub stars with 96 added today, indicating active community traction.