simular-ai/Agent-S
Agent S is an open-source framework that enables autonomous agents to interact with computers through GUIs, leveraging large language models and web retrieval to perform complex tasks with state-of-the-art performance.
Awesome AI Agents › Visual GUI Agents
xlang-ai/aguvis
AGUVIS is a cutting-edge framework designed for autonomous GUI agents that operate purely based on vision, without relying on textual or closed-source models. It supports cross-platform GUI interaction, including web, desktop, and mobile environments, by leveraging a unified action space and plugin system. This approach enables better generalization and consistent performance across different platforms. The project introduces a novel two-stage training pipeline: the first stage focuses on GUI grounding, where the agent learns to understand and interpret visual elements of the interface, and the second stage emphasizes planning and reasoning to execute tasks effectively. AGUVIS also incorporates an inner monologue mechanism, allowing the model to explicitly plan and reason during task execution, enhancing its decision-making capabilities. The framework is supported by a comprehensive large-scale dataset of GUI agent trajectories, which includes multimodal grounding and reasoning data, facilitating robust training and evaluation. AGUVIS demonstrates state-of-the-art performance in both offline and real-world online scenarios, making it a significant advancement in GUI automation technology. The project provides detailed instructions for installation, data preparation, training, and inference, making it accessible for researchers and developers interested in autonomous GUI interaction. Model checkpoints and datasets are available through Hugging Face, supporting reproducibility and further experimentation. Overall, AGUVIS represents a unified, vision-based solution for autonomous GUI agents, pushing the boundaries of what is possible in automated GUI interaction across diverse platforms.
https://github.com/xlang-ai/aguvis
Also in Visual GUI Agents
TencentQQGYLab/AppAgent
AppAgent is an LLM-based multimodal agent framework that enables AI agents to operate smartphone applications through human-like interactions such as tapping and swiping without backend access.
showlab/ShowUI
ShowUI is an open-source, end-to-end vision-language-action model designed for intelligent GUI agents and computer interaction, featuring advanced training, inference, and deployment capabilities.
agno-agi/agent-ui
Agent UI is a modern, customizable chat interface for AI agents featuring real-time streaming, tool call visualization, reasoning steps, and multi-modal content support, built with Next.js and Tailwind CSS.
zai-org/CogAgent
CogAgent is an advanced open-source vision-language model-based GUI agent supporting bilingual interaction and excelling in GUI perception, reasoning, and task generalization.
Ayush0Chaudhary/blurr
Panda is an on-device Android agent written in Kotlin that reads the screen through the Accessibility Service and drives app interfaces with taps, swipes and typing to carry out natural language commands.
likaixin2000/ScreenSpot-Pro-GUI-Grounding
ScreenSpot-Pro is a comprehensive project providing tools, datasets, and benchmarks for GUI grounding tailored to professional high-resolution computer use, advancing AI-driven human-computer interaction.
OSU-NLP-Group/UGround
UGround is a universal visual grounding project for GUI agents enabling AI to navigate and interact with digital interfaces as humans do, achieving state-of-the-art results on multiple benchmarks.