Spider 2.0-AIFunc

Spider 2.0-AIFunc benchmark evaluates text-to-SQL for AI-native SQL workflows on Snowflake

Researchers introduce Spider 2.0-AIFunc, a benchmark of 465 verified instances across 125 real-world databases targeting AI-native SQL functions (classification, sentiment analysis, similarity search, etc.) natively exposed by cloud platforms like Snowflake. Existing text-to-SQL benchmarks cover only conventional SQL, leaving a gap in evaluating whether models can generate queries that invoke LLM capabilities inline. Evaluation of ten state-of-the-art models finds top proprietary models reach 67-70% execution accuracy versus 58.1% for the best open-source model, with failures concentrated in predicate specification, schema grounding, and AI function parameterization. Notably, elaborate agent frameworks designed for traditional text-to-SQL do not transfer well to this setting, with minimal agent setups matching or outperforming more complex alternatives.