mm-toolsandbox-08b0815f·1 events·first seen Aliases: MM-ToolSandBox
Apple researchers introduce MM-ToolSandBox, a benchmark and evaluation framework for visually grounded tool-calling agents, covering 500+ tools across 16 domains with 258 human-verified scenarios. Evaluating 12 models from 4B open-weight to frontier proprietary systems, even the best model achieves below 50% success rate. A key finding is a planning-to-precision crossover: smaller models fail at task planning while larger models fail at visual information extraction, with 53% of failures in capable models stemming from incorrect image interpretation despite correct workflows. The framework and benchmark are publicly released.