simular-ai/Agent-S
Agent S is an open-source framework that enables autonomous agents to interact with computers through GUIs, leveraging large language models and web retrieval to perform complex tasks with state-of-the-art performance.
Awesome AI Agents › Visual GUI Agents
TencentQQGYLab/AppAgent
AppAgent is a cutting-edge multimodal agent framework designed to operate smartphone applications using large language models (LLMs). It enables agents to interact with smartphone apps through human-like actions such as tapping and swiping, without requiring backend system access. This approach allows the agent to work across a wide range of applications, enhancing its versatility and applicability. The framework incorporates an innovative learning method where the agent can learn to navigate and use new apps either autonomously or by observing human demonstrations. This learning process helps the agent build a knowledge base that it can reference to perform complex tasks across different apps. The project supports multi-modal models like GPT-4V and qwen-vl-max, allowing the agent to process both text and visual inputs for decision-making. Users can deploy the agent on real Android devices or emulators, with detailed setup instructions provided, including prerequisites like Android Debug Bridge (adb) and USB debugging. The project is open-source and actively maintained, with ongoing updates and enhancements such as the release of AppAgentX, a next-generation GUI agent with evolving mechanisms. The repository includes demos showcasing the agent's capabilities, such as following users on social media, passing CAPTCHA tests, and using grid overlays for UI element interaction. AppAgent is designed to be accessible, with configuration options for different models and usage scenarios, making it a powerful tool for automating smartphone app interactions through AI-driven agents.
https://github.com/TencentQQGYLab/AppAgent
Also in Visual GUI Agents
showlab/ShowUI
ShowUI is an open-source, end-to-end vision-language-action model designed for intelligent GUI agents and computer interaction, featuring advanced training, inference, and deployment capabilities.
agno-agi/agent-ui
Agent UI is a modern, customizable chat interface for AI agents featuring real-time streaming, tool call visualization, reasoning steps, and multi-modal content support, built with Next.js and Tailwind CSS.
zai-org/CogAgent
CogAgent is an advanced open-source vision-language model-based GUI agent supporting bilingual interaction and excelling in GUI perception, reasoning, and task generalization.
Ayush0Chaudhary/blurr
Panda is an on-device Android agent written in Kotlin that reads the screen through the Accessibility Service and drives app interfaces with taps, swipes and typing to carry out natural language commands.
likaixin2000/ScreenSpot-Pro-GUI-Grounding
ScreenSpot-Pro is a comprehensive project providing tools, datasets, and benchmarks for GUI grounding tailored to professional high-resolution computer use, advancing AI-driven human-computer interaction.
xlang-ai/aguvis
AGUVIS is a unified pure vision-based framework for autonomous GUI agents that operate across multiple platforms, leveraging a novel two-stage training pipeline and inner monologue for enhanced planning and reasoning.
OSU-NLP-Group/UGround
UGround is a universal visual grounding project for GUI agents enabling AI to navigate and interact with digital interfaces as humans do, achieving state-of-the-art results on multiple benchmarks.