wandb/openui
OpenUI is an open-source tool that enables users to describe user interfaces using natural language and see them rendered live, leveraging large language models to generate UI components across multiple frontend frameworks.
Awesome AI Agents › GUI Action Mapping
RUCBM/GUICourse
GUICourse is a comprehensive project aimed at advancing the capabilities of general vision language models (VLMs) to develop versatile graphical user interface (GUI) agents. The project provides datasets, code, and models designed to train and evaluate GUI agents that can understand and interact with GUI environments effectively. The core of GUICourse consists of three major datasets: GUIEnv, GUIAct, and GUIChat. GUIEnv is a large-scale dataset focused on enhancing VLMs' optical character recognition (OCR) and grounding abilities, containing 10 million website page-annotation pairs for pre-training and 0.7 million region-text question-answer pairs for supervised fine-tuning. GUIAct is a GUI navigation dataset that includes 67,000 single-step and 15,000 multi-step action instructions across website and Android scenarios, aimed at improving VLMs' understanding of GUI systems. GUIChat is a conversational dataset with 44,000 single-turn question-answer pairs and 6,000 multi-turn dialogues enriched with text and bounding box information to improve interaction skills of GUI agents. The project offers tools for data preprocessing, supervised fine-tuning, inference, and evaluation, along with released model weights for Qwen-GUI and MiniCPM-GUI. The datasets are available in JSON and parquet formats and can be visualized using provided functions. GUICourse also includes evaluation scripts to assess the performance of GUI agents on various tasks. The project supports interactive testing in simulated Android environments to demonstrate the robustness of the trained agents. Licensed under Creative Commons Attribution 4.0 for datasets and MIT License for code, GUICourse is intended for research purposes to foster the development of intelligent GUI agents capable of understanding and interacting with complex visual interfaces.
https://github.com/RUCBM/GUICourse
Also in GUI Action Mapping
mobile-next/mobile-mcp
Mobile Next MCP is a platform-agnostic Model Context Protocol server enabling scalable mobile automation and interaction with iOS and Android devices, simulators, and emulators through accessibility snapshots and coordinate-based controls.
ddupont808/GPT-4V-Act
GPT-4V-Act is an AI agent that uses GPT-4V(ision) to interact with web user interfaces through mouse and keyboard inputs, enabling enhanced accessibility and automation.
naver/egjs
egjs is a modular collection of JavaScript components designed to simplify and accelerate the development of customizable web applications with a focus on ease of use and performance.
njucckevin/SeeClick
SeeClick is a state-of-the-art project providing models, data, and code for advanced visual GUI agents with a focus on GUI grounding, featuring the ScreenSpot benchmark and superior performance on multi-platform GUI element prediction.
OS-Copilot/OS-Atlas
OS-Atlas is a foundation action model for generalist GUI agents that enables precise visual grounding and interaction with UI elements in screenshots using advanced vision-language models.
AriaUI/Aria-UI
Aria-UI is an open-source, fast, and context-aware action grounding system for GUI instructions, achieving state-of-the-art performance in dynamic agent tasks like AndroidWorld and OSWorld.
opendilab/awesome-ui-agents
A curated and continuously updated collection of research papers, models, tools, and datasets focused on UI agents that interact with various user interfaces across platforms like web, mobile, and OS environments.