TencentQQGYLab/AppAgent
AppAgent is an LLM-based multimodal agent framework that enables AI agents to operate smartphone applications through human-like interactions such as tapping and swiping without backend access.
Awesome AI Agents › Visual GUI Agents
simular-ai/Agent-S
Agent S is an open-source framework designed to create intelligent autonomous agents that interact with computers in a human-like manner through graphical user interfaces (GUIs). The project aims to build computer use agents (CUA) that can learn from past experiences and perform complex tasks autonomously on various operating systems including Mac, Windows, Linux, and Android. Agent S2, the latest version, introduces a compositional generalist-specialist framework that significantly improves performance benchmarks on tasks involving computer interaction, surpassing previous state-of-the-art models such as OpenAI's CUA/Operator and Anthropic's Claude 3.7 Sonnet Computer-Use. The framework supports integration with large language models (LLMs) and visual grounding models like UI-TARS, which can be hosted locally or accessed via Hugging Face Inference Endpoints. It also supports multiple API providers including OpenAI, Anthropic, Gemini, Open Router, and vLLM for inference. Agent S incorporates a web-knowledge retrieval system through Perplexica, a search engine API that enhances the agent's ability to access and utilize web-based information dynamically. Installation involves setting up the gui-agents Python package and configuring environment variables for API keys. The project provides detailed instructions for setting up Perplexica with Docker to enable web retrieval capabilities. Agent S2 has demonstrated improved success rates on benchmarks such as OSWorld, WindowsAgentArena, and AndroidWorld, showcasing its ability to perform multi-step tasks with higher accuracy than previous models. The project is well-documented with technical papers, blog posts, and videos available for both Agent S and Agent S2. It encourages contributions from the AI and automation communities and aims to push the boundaries of autonomous agent-based systems for computer use. Overall, Agent S represents a significant advancement in the development of autonomous agents capable of complex, human-like interaction with computer systems.
https://github.com/simular-ai/Agent-S
Also in Visual GUI Agents
showlab/ShowUI
ShowUI is an open-source, end-to-end vision-language-action model designed for intelligent GUI agents and computer interaction, featuring advanced training, inference, and deployment capabilities.
agno-agi/agent-ui
Agent UI is a modern, customizable chat interface for AI agents featuring real-time streaming, tool call visualization, reasoning steps, and multi-modal content support, built with Next.js and Tailwind CSS.
zai-org/CogAgent
CogAgent is an advanced open-source vision-language model-based GUI agent supporting bilingual interaction and excelling in GUI perception, reasoning, and task generalization.
Ayush0Chaudhary/blurr
Panda is an on-device Android agent written in Kotlin that reads the screen through the Accessibility Service and drives app interfaces with taps, swipes and typing to carry out natural language commands.
likaixin2000/ScreenSpot-Pro-GUI-Grounding
ScreenSpot-Pro is a comprehensive project providing tools, datasets, and benchmarks for GUI grounding tailored to professional high-resolution computer use, advancing AI-driven human-computer interaction.
xlang-ai/aguvis
AGUVIS is a unified pure vision-based framework for autonomous GUI agents that operate across multiple platforms, leveraging a novel two-stage training pipeline and inner monologue for enhanced planning and reasoning.
OSU-NLP-Group/UGround
UGround is a universal visual grounding project for GUI agents enabling AI to navigate and interact with digital interfaces as humans do, achieving state-of-the-art results on multiple benchmarks.