simular-ai/Agent-S
Agent S is an open-source framework that enables autonomous agents to interact with computers through GUIs, leveraging large language models and web retrieval to perform complex tasks with state-of-the-art performance.
Awesome AI Agents › Visual GUI Agents
zai-org/CogAgent
CogAgent is an open-sourced end-to-end Vision-Language Model (VLM) based GUI Agent designed to interact with graphical user interfaces through both visual and natural language inputs. The latest version, CogAgent-9B-20241220, released in December 2024, introduces significant improvements in GUI perception, reasoning accuracy, action space completeness, task universality, and generalization. It supports bilingual interaction in Chinese and English, enabling users to communicate with the agent using screenshots and natural language commands. CogAgent was recognized as a highlight at CVPR 2024, underscoring its state-of-the-art capabilities in GUI agent tasks. The model is based on GLM-4V-9B, a bilingual open-source VLM base model, and has undergone multi-stage training and optimization to enhance its performance. CogAgent excels in GUI localization, single-step and multi-step operations, and Chinese step-wise benchmarks, outperforming several commercial and open-source models. It is integrated into ZhipuAI's GLM-PC product and is available for download and demo through platforms like HuggingFace, ModelScope, and WiseModel. The project provides comprehensive documentation, including technical blogs, practical guides, and an online demo. The model requires substantial GPU resources for inference and fine-tuning, with recommendations for VRAM usage and precision settings. CogAgent aims to advance research and applications in GUI agents by providing a powerful, versatile, and accessible tool for developers and researchers working with vision-language models in GUI environments.
https://github.com/zai-org/CogAgent
Also in Visual GUI Agents
TencentQQGYLab/AppAgent
AppAgent is an LLM-based multimodal agent framework that enables AI agents to operate smartphone applications through human-like interactions such as tapping and swiping without backend access.
showlab/ShowUI
ShowUI is an open-source, end-to-end vision-language-action model designed for intelligent GUI agents and computer interaction, featuring advanced training, inference, and deployment capabilities.
agno-agi/agent-ui
Agent UI is a modern, customizable chat interface for AI agents featuring real-time streaming, tool call visualization, reasoning steps, and multi-modal content support, built with Next.js and Tailwind CSS.
Ayush0Chaudhary/blurr
Panda is an on-device Android agent written in Kotlin that reads the screen through the Accessibility Service and drives app interfaces with taps, swipes and typing to carry out natural language commands.
likaixin2000/ScreenSpot-Pro-GUI-Grounding
ScreenSpot-Pro is a comprehensive project providing tools, datasets, and benchmarks for GUI grounding tailored to professional high-resolution computer use, advancing AI-driven human-computer interaction.
xlang-ai/aguvis
AGUVIS is a unified pure vision-based framework for autonomous GUI agents that operate across multiple platforms, leveraging a novel two-stage training pipeline and inner monologue for enhanced planning and reasoning.
OSU-NLP-Group/UGround
UGround is a universal visual grounding project for GUI agents enabling AI to navigate and interact with digital interfaces as humans do, achieving state-of-the-art results on multiple benchmarks.