simular-ai/Agent-S
Agent S is an open-source framework that enables autonomous agents to interact with computers through GUIs, leveraging large language models and web retrieval to perform complex tasks with state-of-the-art performance.
Awesome AI Agents › Visual GUI Agents
eric-ai-lab/Screen-Point-and-Read
The project "Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding" addresses the challenge of screen reading based on user-indicated points on graphical user interfaces (GUIs). GUIs are essential for interacting with digital devices, and while many models exist for GUI understanding, the specific task of screen reading from a pointed location has been largely overlooked. This project introduces the Screen Point-and-Read (ScreenPR) task and proposes a novel Tree-of-Lens (ToL) agent to tackle it. The ToL agent uses a unique grounding mechanism that constructs a Hierarchical Layout Tree from the input point coordinate and the corresponding GUI screenshot. This tree structure allows the agent to comprehend not only the content within the indicated area but also the layout and spatial relationships between GUI elements, which is critical for accurate interpretation. The project includes training a GUI region detection model fine-tuned on the DINO detection model with MMDetection, which detects local and global regions in GUI screenshots. The ToL agent pipeline involves constructing the Hierarchical Layout Tree, selecting the target path based on the input point, and generating prompts using multiple lenses. The project also provides evaluation scripts and benchmarks, including the ScreenPR benchmark with GUIs from mobile, web, and operating systems. Additionally, the ToL agent is applied to verify mobile GUI navigation tasks by analyzing agent trajectories to identify incorrect actions. This verification process is demonstrated using the MagicWonder agent on the MagicWand platform. The project offers datasets, training instructions, and detailed guides for setup and usage. Overall, this project advances the field of GUI understanding by introducing a layout-aware screen reading tool that leverages hierarchical layout information for improved accuracy and utility in real-world applications such as mobile navigation verification.
https://github.com/eric-ai-lab/Screen-Point-and-Read
Also in Visual GUI Agents
TencentQQGYLab/AppAgent
AppAgent is an LLM-based multimodal agent framework that enables AI agents to operate smartphone applications through human-like interactions such as tapping and swiping without backend access.
showlab/ShowUI
ShowUI is an open-source, end-to-end vision-language-action model designed for intelligent GUI agents and computer interaction, featuring advanced training, inference, and deployment capabilities.
agno-agi/agent-ui
Agent UI is a modern, customizable chat interface for AI agents featuring real-time streaming, tool call visualization, reasoning steps, and multi-modal content support, built with Next.js and Tailwind CSS.
zai-org/CogAgent
CogAgent is an advanced open-source vision-language model-based GUI agent supporting bilingual interaction and excelling in GUI perception, reasoning, and task generalization.
Ayush0Chaudhary/blurr
Panda is an on-device Android agent written in Kotlin that reads the screen through the Accessibility Service and drives app interfaces with taps, swipes and typing to carry out natural language commands.
likaixin2000/ScreenSpot-Pro-GUI-Grounding
ScreenSpot-Pro is a comprehensive project providing tools, datasets, and benchmarks for GUI grounding tailored to professional high-resolution computer use, advancing AI-driven human-computer interaction.
xlang-ai/aguvis
AGUVIS is a unified pure vision-based framework for autonomous GUI agents that operate across multiple platforms, leveraging a novel two-stage training pipeline and inner monologue for enhanced planning and reasoning.