spider-2-0-b599f8b2·2 events·first seen Aliases: Spider 2.0
Researchers introduce Spider 2.0-AIFunc, a benchmark of 465 verified instances across 125 real-world databases targeting AI-native SQL functions (classification, sentiment analysis, similarity search, etc.) natively exposed by cloud platforms like Snowflake. Existing text-to-SQL benchmarks cover only conventional SQL, leaving a gap in evaluating whether models can generate queries that invoke LLM capabilities inline. Evaluation of ten state-of-the-art models finds top proprietary models reach 67-70% execution accuracy versus 58.1% for the best open-source model, with failures concentrated in predicate specification, schema grounding, and AI function parameterization. Notably, elaborate agent frameworks designed for traditional text-to-SQL do not transfer well to this setting, with minimal agent setups matching or outperforming more complex alternatives.
TAHOE is a Text-to-SQL system that treats prompt optimization as a dynamic data management problem, building a structured Hint Bank from compiler, execution, and user feedback without updating model parameters. On the Spider 2.0-Snow benchmark using GPT-5.5, it raises pass rate from 61.95% to 79.42% and achieves 100% Snowflake syntax compliance while reducing compiler-feedback rounds from 2.79 to 0.12. The learned Hint Bank transfers to weaker models, yielding a 19.7 percentage-point gain on Doubao-2.0-lite. The approach targets the production deployment gap between Text-to-SQL prototypes and real-world database environments with strict dialects and large schemas.