wandb/openui
OpenUI is an open-source tool that enables users to describe user interfaces using natural language and see them rendered live, leveraging large language models to generate UI components across multiple frontend frameworks.
Awesome AI Agents › GUI Action Mapping
deepneuralmachine/seq2act-tensorflow
Seq2act is a research project developed by Google researchers that focuses on mapping natural language instructions to sequences of actions on mobile user interfaces (UI). The project provides code and an experimental framework to train and evaluate models that can understand and convert natural language commands into actionable UI sequences. This capability is particularly useful for automating interactions with mobile applications based on verbal or written instructions, enhancing accessibility and user experience. The repository includes implementations for two main types of models: Phrase Tuple Extraction models and Grounding models. Phrase Tuple Extraction models are designed to parse and extract meaningful components from natural language instructions, while Grounding models link these extracted phrases to specific UI actions on mobile devices. The framework supports training, continuous evaluation, and testing of these models, allowing researchers and developers to experiment with and improve the system. The project was presented at the 2020 Annual Conference of the Association for Computational Linguistics (ACL), highlighting its significance in the field of natural language processing and human-computer interaction. Although the datasets and data pipelines are planned for future updates, the current codebase allows users to run minimal versions of the models and conduct experiments using provided scripts. Seq2act's approach can be applied to various applications, including voice-controlled mobile interfaces, automated testing of mobile apps, and assistive technologies for users with disabilities. By bridging the gap between natural language and mobile UI actions, this project contributes to advancing intelligent user interfaces and improving the way users interact with mobile technology.
https://github.com/deepneuralmachine/seq2act-tensorflow
Also in GUI Action Mapping
mobile-next/mobile-mcp
Mobile Next MCP is a platform-agnostic Model Context Protocol server enabling scalable mobile automation and interaction with iOS and Android devices, simulators, and emulators through accessibility snapshots and coordinate-based controls.
ddupont808/GPT-4V-Act
GPT-4V-Act is an AI agent that uses GPT-4V(ision) to interact with web user interfaces through mouse and keyboard inputs, enabling enhanced accessibility and automation.
naver/egjs
egjs is a modular collection of JavaScript components designed to simplify and accelerate the development of customizable web applications with a focus on ease of use and performance.
njucckevin/SeeClick
SeeClick is a state-of-the-art project providing models, data, and code for advanced visual GUI agents with a focus on GUI grounding, featuring the ScreenSpot benchmark and superior performance on multi-platform GUI element prediction.
OS-Copilot/OS-Atlas
OS-Atlas is a foundation action model for generalist GUI agents that enables precise visual grounding and interaction with UI elements in screenshots using advanced vision-language models.
AriaUI/Aria-UI
Aria-UI is an open-source, fast, and context-aware action grounding system for GUI instructions, achieving state-of-the-art performance in dynamic agent tasks like AndroidWorld and OSWorld.
opendilab/awesome-ui-agents
A curated and continuously updated collection of research papers, models, tools, and datasets focused on UI agents that interact with various user interfaces across platforms like web, mobile, and OS environments.