simular-ai/Agent-S
Agent S is an open-source framework that enables autonomous agents to interact with computers through GUIs, leveraging large language models and web retrieval to perform complex tasks with state-of-the-art performance.
Awesome AI Agents › Visual GUI Agents
OSU-NLP-Group/UGround
UGround is a cutting-edge project focused on universal visual grounding for GUI (Graphical User Interface) agents, presented as an oral paper at ICLR 2025. This project is a collaboration between the OSU NLP Group and Orby AI, aiming to enable AI agents to navigate digital interfaces as humans do by understanding and grounding visual elements in GUIs. The repository provides comprehensive resources including model weights, training data, code for training and inference, offline and online experiment setups, and a live demo. UGround leverages advanced multimodal models, including versions based on Qwen2-VL, to achieve state-of-the-art performance on various GUI grounding benchmarks such as ScreenSpot and ScreenSpot-Pro. The project includes a detailed evaluation suite to facilitate testing and benchmarking of GUI agents and grounding models. The repository also hosts multiple offline evaluation tools and online experiment frameworks, supporting research and development in GUI interaction and automation. The main results demonstrate UGround's superior performance compared to other models, highlighting its effectiveness in understanding and interacting with diverse GUI elements across mobile, desktop, and web platforms. The project is actively maintained with ongoing updates, including the release of training data, model improvements, and expanded evaluation capabilities. UGround represents a significant advancement in the field of human-computer interaction, providing a universal solution for visual grounding in GUI environments, which is crucial for developing intelligent agents capable of seamless digital navigation and interaction.
https://github.com/OSU-NLP-Group/UGround
Also in Visual GUI Agents
TencentQQGYLab/AppAgent
AppAgent is an LLM-based multimodal agent framework that enables AI agents to operate smartphone applications through human-like interactions such as tapping and swiping without backend access.
showlab/ShowUI
ShowUI is an open-source, end-to-end vision-language-action model designed for intelligent GUI agents and computer interaction, featuring advanced training, inference, and deployment capabilities.
agno-agi/agent-ui
Agent UI is a modern, customizable chat interface for AI agents featuring real-time streaming, tool call visualization, reasoning steps, and multi-modal content support, built with Next.js and Tailwind CSS.
zai-org/CogAgent
CogAgent is an advanced open-source vision-language model-based GUI agent supporting bilingual interaction and excelling in GUI perception, reasoning, and task generalization.
Ayush0Chaudhary/blurr
Panda is an on-device Android agent written in Kotlin that reads the screen through the Accessibility Service and drives app interfaces with taps, swipes and typing to carry out natural language commands.
likaixin2000/ScreenSpot-Pro-GUI-Grounding
ScreenSpot-Pro is a comprehensive project providing tools, datasets, and benchmarks for GUI grounding tailored to professional high-resolution computer use, advancing AI-driven human-computer interaction.
xlang-ai/aguvis
AGUVIS is a unified pure vision-based framework for autonomous GUI agents that operate across multiple platforms, leveraging a novel two-stage training pipeline and inner monologue for enhanced planning and reasoning.