simular-ai/Agent-S
Agent S is an open-source framework that enables autonomous agents to interact with computers through GUIs, leveraging large language models and web retrieval to perform complex tasks with state-of-the-art performance.
Awesome AI Agents › Visual GUI Agents
likaixin2000/ScreenSpot-Pro-GUI-Grounding
ScreenSpot-Pro is a project focused on GUI grounding tailored for professional high-resolution computer use. It aims to enhance the interaction and understanding of graphical user interfaces (GUIs) in environments where high-resolution displays are prevalent, which is common in professional settings. The project provides tools and datasets to facilitate the evaluation and development of models that can interpret and interact with complex GUI elements effectively. It is supported by a research paper that details the methodologies and findings related to GUI grounding, making it a valuable resource for researchers and developers working on advanced human-computer interaction and AI-driven GUI understanding. The repository includes scripts for setting up the environment, particularly requiring an OpenAI API key, indicating integration with advanced language models or AI services. It also provides shell scripts to run evaluations, suggesting that the project includes benchmarking capabilities to assess the performance of GUI grounding models. The project is actively maintained and has been recognized and used as a benchmark in several notable projects, highlighting its relevance and impact in the field. ScreenSpot-Pro offers a dataset hosted on Huggingface, which is a popular platform for sharing machine learning datasets, further supporting its utility for training and testing AI models in GUI grounding tasks. The project encourages contributions and collaboration, as indicated by the welcoming badge for contributions. It also maintains a leaderboard, which likely tracks the performance of different models or approaches on the GUI grounding tasks defined by the project. Overall, ScreenSpot-Pro serves as a comprehensive framework and benchmark for advancing the state-of-the-art in GUI grounding, particularly for professional applications involving high-resolution displays. It combines research, datasets, evaluation tools, and community engagement to foster innovation and progress in this specialized area of computer vision and human-computer interaction.
https://github.com/likaixin2000/ScreenSpot-Pro-GUI-Grounding
Also in Visual GUI Agents
TencentQQGYLab/AppAgent
AppAgent is an LLM-based multimodal agent framework that enables AI agents to operate smartphone applications through human-like interactions such as tapping and swiping without backend access.
showlab/ShowUI
ShowUI is an open-source, end-to-end vision-language-action model designed for intelligent GUI agents and computer interaction, featuring advanced training, inference, and deployment capabilities.
agno-agi/agent-ui
Agent UI is a modern, customizable chat interface for AI agents featuring real-time streaming, tool call visualization, reasoning steps, and multi-modal content support, built with Next.js and Tailwind CSS.
zai-org/CogAgent
CogAgent is an advanced open-source vision-language model-based GUI agent supporting bilingual interaction and excelling in GUI perception, reasoning, and task generalization.
Ayush0Chaudhary/blurr
Panda is an on-device Android agent written in Kotlin that reads the screen through the Accessibility Service and drives app interfaces with taps, swipes and typing to carry out natural language commands.
xlang-ai/aguvis
AGUVIS is a unified pure vision-based framework for autonomous GUI agents that operate across multiple platforms, leveraging a novel two-stage training pipeline and inner monologue for enhanced planning and reasoning.
OSU-NLP-Group/UGround
UGround is a universal visual grounding project for GUI agents enabling AI to navigate and interact with digital interfaces as humans do, achieving state-of-the-art results on multiple benchmarks.