devicesworld-8a8ad2a1·1 events·first seen Aliases: DevicesWorld
Researchers introduce DevicesWorld, a large-scale benchmark of 6,140 tasks designed to evaluate LLM-based agents operating across heterogeneous device environments including mobile, desktop, and IoT systems. The benchmark addresses a gap in existing evaluations that focus on single-device settings, instead requiring agents to acquire and integrate information across devices to complete end-to-end tasks. Five frontier LLM-agent systems were evaluated, with the best achieving only 12.5% success rate, revealing systematic failure modes including device confusion and premature termination. The work provides an executable, automatically verifiable evaluation framework for cross-device agent research.