GROW²

GROW² framework enables open-world affordance grounding for creative robot tool use

Researchers introduce GROW² (GROunding Which and Where), a hierarchical framework for robot tool use that enables selecting and localizing arbitrary objects as tools beyond their intended functions. The system uses Vision-Language Models for semantic reasoning to identify suitable tools and task-relevant parts, then vision foundation models for geometric grounding into 3D regions from RGB-D images. GROW² achieves zero-shot generalization over open-category objects and outperforms baselines on affordance prediction benchmarks in both simulated and real-world settings, without requiring end-to-end training on large datasets.