simular-ai/Agent-S
Agent S is an open-source framework that enables autonomous agents to interact with computers through GUIs, leveraging large language models and web retrieval to perform complex tasks with state-of-the-art performance.
Awesome AI Agents › Visual GUI Agents
showlab/ShowUI
ShowUI is an open-source, end-to-end, lightweight vision-language-action model designed specifically for GUI agents and computer use. It integrates visual perception, language understanding, and action execution into a single unified model, enabling intelligent interaction with graphical user interfaces. The project is developed by Show Lab at the National University of Singapore in collaboration with Microsoft and has been recognized with an Outstanding Paper Award at NeurIPS 2024 Open-World Agents workshop and accepted for presentation at CVPR 2025. The model supports various advanced features including fine-tuning and inference with the latest Qwen2.5-VL base model, vllm inference for efficient multi-GPU usage, and API calling through a Gradio client for easy deployment without requiring a GPU. ShowUI also supports navigation tasks such as Mind2Web, AITW, and Miniwob, and offers iterative refinement to improve grounding accuracy. The project provides datasets like ShowUI-web and ShowUI-desktop, along with training codebases that support grounding and navigation training, efficient training techniques like DeepSpeed, BF16, QLoRA, and multi-GPU/multi-node setups. ShowUI is integrated into the Computer Use OOTB project for local execution, allowing users to control their PCs using the model. It also features UI-guided token selection and tools for annotating data with GPT-4o. The repository includes comprehensive documentation, demos, and examples to facilitate usage and experimentation. The project is well-supported with links to papers, datasets, demos, and training resources, making it a valuable tool for researchers and developers working on vision-language-action models and intelligent GUI agents.
https://github.com/showlab/ShowUI
Also in Visual GUI Agents
TencentQQGYLab/AppAgent
AppAgent is an LLM-based multimodal agent framework that enables AI agents to operate smartphone applications through human-like interactions such as tapping and swiping without backend access.
agno-agi/agent-ui
Agent UI is a modern, customizable chat interface for AI agents featuring real-time streaming, tool call visualization, reasoning steps, and multi-modal content support, built with Next.js and Tailwind CSS.
zai-org/CogAgent
CogAgent is an advanced open-source vision-language model-based GUI agent supporting bilingual interaction and excelling in GUI perception, reasoning, and task generalization.
Ayush0Chaudhary/blurr
Panda is an on-device Android agent written in Kotlin that reads the screen through the Accessibility Service and drives app interfaces with taps, swipes and typing to carry out natural language commands.
likaixin2000/ScreenSpot-Pro-GUI-Grounding
ScreenSpot-Pro is a comprehensive project providing tools, datasets, and benchmarks for GUI grounding tailored to professional high-resolution computer use, advancing AI-driven human-computer interaction.
xlang-ai/aguvis
AGUVIS is a unified pure vision-based framework for autonomous GUI agents that operate across multiple platforms, leveraging a novel two-stage training pipeline and inner monologue for enhanced planning and reasoning.
OSU-NLP-Group/UGround
UGround is a universal visual grounding project for GUI agents enabling AI to navigate and interact with digital interfaces as humans do, achieving state-of-the-art results on multiple benchmarks.