simular-ai/Agent-S
Agent S is an open-source framework that enables autonomous agents to interact with computers through GUIs, leveraging large language models and web retrieval to perform complex tasks with state-of-the-art performance.
Awesome AI Agents › Visual GUI Agents
Ayush0Chaudhary/blurr
Panda, developed in the blurr repository, is a phone operator agent for Android that takes a natural language instruction and completes it by controlling the device user interface directly, across whichever applications are needed. The architecture separates perception and action from reasoning. The Android Accessibility Service acts as the actuator layer: it exposes the element hierarchy of the current screen and performs programmatic touch gestures, which gives the agent both its eyes and its hands. An LLM supplies the higher level reasoning, planning and analysis, deciding which element to act on next and when a multi-step task is finished. The project describes itself as a multi-agent system written entirely in Kotlin, with an Operator component that executes steps while keeping a notepad of working state. Voice output is provided through a cloud speech synthesis service. A persistent, personalised local memory subsystem exists in the codebase but the README notes it is temporarily disabled pending a future update. Configuration is handled through environment variables: the app can be pointed either at Gemini API keys directly or at an arbitrary proxy endpoint that accepts a documented request payload, which allows any compatible backend to serve as the model provider. Building it requires Android Studio and a device or emulator on API level 26 or above; on first launch the app asks the user to grant Accessibility permission, without which the agent cannot observe or control the screen. The published demonstrations include multi-step tasks such as sending messages to new contacts on a social network. The repository is marked as a work in progress and is distributed under a personal-use licence, with a companion listing on the Google Play store. It is aimed at developers experimenting with mobile GUI automation agents and accessibility-driven device control.
https://github.com/Ayush0Chaudhary/blurr
Also in Visual GUI Agents
TencentQQGYLab/AppAgent
AppAgent is an LLM-based multimodal agent framework that enables AI agents to operate smartphone applications through human-like interactions such as tapping and swiping without backend access.
showlab/ShowUI
ShowUI is an open-source, end-to-end vision-language-action model designed for intelligent GUI agents and computer interaction, featuring advanced training, inference, and deployment capabilities.
agno-agi/agent-ui
Agent UI is a modern, customizable chat interface for AI agents featuring real-time streaming, tool call visualization, reasoning steps, and multi-modal content support, built with Next.js and Tailwind CSS.
zai-org/CogAgent
CogAgent is an advanced open-source vision-language model-based GUI agent supporting bilingual interaction and excelling in GUI perception, reasoning, and task generalization.
likaixin2000/ScreenSpot-Pro-GUI-Grounding
ScreenSpot-Pro is a comprehensive project providing tools, datasets, and benchmarks for GUI grounding tailored to professional high-resolution computer use, advancing AI-driven human-computer interaction.
xlang-ai/aguvis
AGUVIS is a unified pure vision-based framework for autonomous GUI agents that operate across multiple platforms, leveraging a novel two-stage training pipeline and inner monologue for enhanced planning and reasoning.
OSU-NLP-Group/UGround
UGround is a universal visual grounding project for GUI agents enabling AI to navigate and interact with digital interfaces as humans do, achieving state-of-the-art results on multiple benchmarks.