wandb/openui
OpenUI is an open-source tool that enables users to describe user interfaces using natural language and see them rendered live, leveraging large language models to generate UI components across multiple frontend frameworks.
Awesome AI Agents › GUI Action Mapping
opendilab/awesome-ui-agents
The project "Awesome UI Agent" is a curated and continuously updated collection of resources related to UI agents, which are intelligent agents designed to interact with various user interfaces across different platforms such as web, mobile apps, and operating systems. The repository compiles research papers, models, tools, and datasets that focus on the development and advancement of UI agents. These agents leverage vision-language models to understand and interact with user interfaces, enabling them to perform tasks like mobile device operation, web browsing, and game playing. The project highlights the interdisciplinary nature of UI agent research, involving fields such as computer vision, natural language processing, reinforcement learning, human-computer interaction, and software engineering. The repository categorizes resources by year and type, including recent and upcoming research papers from 2023 to 2025, covering topics like reinforcement learning frameworks, vision-language models, distributed training, and multi-modal agent tuning. It also features notable projects from organizations like OpenAI and Anthropic, showcasing agents capable of using browsers or emulating human-computer interactions. The collection includes benchmarks and experimental environments that facilitate the evaluation of UI agents based on task completion, efficiency, and generalization. The project serves as a valuable resource for researchers, developers, and enthusiasts interested in the frontier of UI agent technology. It provides insights into the challenges faced by the field, such as scalability, robustness, and interpretability of agents. By aggregating state-of-the-art research and tools, the repository aims to foster collaboration and innovation in creating generalist agents capable of seamless interaction with diverse digital environments. The repository also encourages community contributions to keep the content up-to-date and comprehensive.
https://github.com/opendilab/awesome-ui-agents
Also in GUI Action Mapping
mobile-next/mobile-mcp
Mobile Next MCP is a platform-agnostic Model Context Protocol server enabling scalable mobile automation and interaction with iOS and Android devices, simulators, and emulators through accessibility snapshots and coordinate-based controls.
ddupont808/GPT-4V-Act
GPT-4V-Act is an AI agent that uses GPT-4V(ision) to interact with web user interfaces through mouse and keyboard inputs, enabling enhanced accessibility and automation.
naver/egjs
egjs is a modular collection of JavaScript components designed to simplify and accelerate the development of customizable web applications with a focus on ease of use and performance.
njucckevin/SeeClick
SeeClick is a state-of-the-art project providing models, data, and code for advanced visual GUI agents with a focus on GUI grounding, featuring the ScreenSpot benchmark and superior performance on multi-platform GUI element prediction.
OS-Copilot/OS-Atlas
OS-Atlas is a foundation action model for generalist GUI agents that enables precise visual grounding and interaction with UI elements in screenshots using advanced vision-language models.
AriaUI/Aria-UI
Aria-UI is an open-source, fast, and context-aware action grounding system for GUI instructions, achieving state-of-the-art performance in dynamic agent tasks like AndroidWorld and OSWorld.
cooelf/Auto-GUI
Auto-GUI is a multimodal AI agent framework that predicts user interface actions using a novel chain-of-action technique, enabling direct interaction with interfaces without environment parsing or application-specific APIs.