wandb/openui
OpenUI is an open-source tool that enables users to describe user interfaces using natural language and see them rendered live, leveraging large language models to generate UI components across multiple frontend frameworks.
Awesome AI Agents › GUI Action Mapping
njucckevin/SeeClick
SeeClick is a comprehensive project that provides the model, data, and code for advanced visual GUI agents, focusing on GUI grounding. It is based on the paper "SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents" and includes a GUI grounding benchmark called ScreenSpot, which contains over 1200 instructions from various platforms such as iOS, Android, macOS, Windows, and Web. The benchmark includes annotated element types like Text or Icon/Widget, making it a valuable resource for evaluating GUI grounding performance. The project offers a large-scale web GUI grounding corpus collected from Common Crawl for pre-training purposes. SeeClick is built on the Qwen-VL model and is compatible with its Transformers inference code, allowing users to easily run inference with minimal code. The project also provides pre-training and evaluation scripts, fine-tuning guidelines using LoRA, and downstream agent task training and testing instructions. The model demonstrates superior performance on the ScreenSpot benchmark compared to other large vision-language models, particularly in predicting click points on GUIs. The repository includes detailed instructions for setting up the environment, running inference, fine-tuning, and pre-training, making it accessible for researchers and developers interested in GUI grounding and visual GUI agents. Overall, SeeClick is a state-of-the-art tool for advancing the capabilities of visual GUI agents through effective GUI grounding techniques and extensive datasets.
https://github.com/njucckevin/SeeClick
Also in GUI Action Mapping
mobile-next/mobile-mcp
Mobile Next MCP is a platform-agnostic Model Context Protocol server enabling scalable mobile automation and interaction with iOS and Android devices, simulators, and emulators through accessibility snapshots and coordinate-based controls.
ddupont808/GPT-4V-Act
GPT-4V-Act is an AI agent that uses GPT-4V(ision) to interact with web user interfaces through mouse and keyboard inputs, enabling enhanced accessibility and automation.
naver/egjs
egjs is a modular collection of JavaScript components designed to simplify and accelerate the development of customizable web applications with a focus on ease of use and performance.
OS-Copilot/OS-Atlas
OS-Atlas is a foundation action model for generalist GUI agents that enables precise visual grounding and interaction with UI elements in screenshots using advanced vision-language models.
AriaUI/Aria-UI
Aria-UI is an open-source, fast, and context-aware action grounding system for GUI instructions, achieving state-of-the-art performance in dynamic agent tasks like AndroidWorld and OSWorld.
opendilab/awesome-ui-agents
A curated and continuously updated collection of research papers, models, tools, and datasets focused on UI agents that interact with various user interfaces across platforms like web, mobile, and OS environments.
cooelf/Auto-GUI
Auto-GUI is a multimodal AI agent framework that predicts user interface actions using a novel chain-of-action technique, enabling direct interaction with interfaces without environment parsing or application-specific APIs.