ScreenSuite: Comprehensive Evaluation Suite for GUI Agents
Hugging Face has released ScreenSuite, described as the most comprehensive evaluation suite for GUI (Graphical User Interface) agents. The suite aims to standardize and broaden benchmarking for agents that interact with visual interfaces. This addresses a gap in the evaluation ecosystem for screen-based AI agents, which are increasingly relevant as agentic systems expand into desktop and web automation tasks.
Related guides (3)
Related events (8)
ScreenEnv: Deploy your full stack Desktop Agent
Hugging Face introduces ScreenEnv, a framework for deploying full-stack desktop agents that can interact with graphical user interfaces. The tool appears to provide infrastructure for building and running computer-use style agents in desktop environments. This is part of the growing ecosystem of agent tooling that enables AI systems to operate software through screen observation and interaction.
VISTA: Hybrid user simulation toolkit for interactive agent evaluation
Researchers introduce VISTA, a user simulation framework designed to address limitations in current agent evaluation methods, which rely on static benchmarks that miss dynamic, multi-step failure modes. VISTA provides six metrics for measuring realism, capability coverage, and interaction effectiveness, and combines UI-based and API-based interactions in a hybrid simulator. The toolkit is evaluated in e-commerce and education customer service settings, showing more realistic and comprehensive coverage than existing approaches.
HiViG: History-aware visually grounded critic improves computer use agents across GUI benchmarks
Researchers introduce HiViG, a test-time framework for Computer Use Agents that addresses two weaknesses in existing critic models: short-sighted decision loops and lack of visual grounding. The system trains a multimodal critic on real GUI trajectories to maintain a compact macro-action history and verify execution coordinates against live screenshots before action execution. Evaluated on web, mobile, and desktop benchmarks, HiViG improves average success rates by 5.8% over the strongest baseline with Qwen3-VL-32B and 9.0% with Gemini-3-Flash, with both history and grounding components shown to be independently necessary.
Holo1: New family of GUI automation VLMs powering GUI agent Surfer-H
H Company has released Holo1, a new family of vision-language models specifically designed for GUI automation tasks. These models power Surfer-H, a GUI agent capable of interacting with graphical interfaces. The release represents a specialized VLM family targeting the agent-tool ecosystem for desktop/web automation. Details on architecture, training data, and benchmarks are expected in the accompanying blog post.
Agent-S: Open Agentic Framework for Human-Like Computer Use
Agent-S is an open-source Python framework by Simular AI designed to enable AI agents to interact with computers in a human-like manner. The project has accumulated 11,388 GitHub stars with modest daily growth of 29 stars. It represents an entry in the growing space of computer-use agent frameworks targeting GUI and desktop automation tasks.
Hugging Face Transformers Code Agent Beats GAIA Benchmark
Hugging Face reports that their Transformers-based code agent has achieved a top score on the GAIA benchmark, a challenging evaluation for general AI assistants requiring multi-step reasoning and tool use. The result positions Hugging Face's open agent framework competitively against proprietary systems. The post details the agent architecture and tooling approach used to achieve the result.
OpenEnv in Practice: Evaluating Tool-Using Agents in Real-World Environments
This Hugging Face blog post introduces OpenEnv, a framework for evaluating tool-using AI agents in real-world environments. The piece appears to address the challenge of benchmarking agentic systems that interact with external tools and environments, moving beyond static benchmarks toward dynamic, practical evaluation settings. As a tier-2 commentary piece, it likely discusses methodology, design choices, and results from applying OpenEnv to assess agent capabilities.
MobileGym: Verifiable Parallel Simulation Platform for Mobile GUI Agent Training
MobileGym is a browser-hosted simulation environment for mobile GUI agent research that enables deterministic outcome verification via structured JSON state and scalable online RL through hundreds of parallel instances (~400 MB/instance, ~3s cold start). The accompanying MobileGym-Bench provides 416 parameterized task templates across 28 apps with deterministic judges. A sim-to-real case study using GRPO on Qwen3-VL-4B-Instruct achieves +12.8 percentage points on the 256-task test set, with real-device execution retaining 95.1% of simulation-side training gains.