wandb/openui
OpenUI is an open-source tool that enables users to describe user interfaces using natural language and see them rendered live, leveraging large language models to generate UI components across multiple frontend frameworks.
Awesome AI Agents › GUI Action Mapping
IMNearth/CoAT
The project "Android in the Zoo: Chain-of-Action-Thought for GUI Agents" (CoAT) presents an innovative approach to improving the interaction of GUI agents with Android applications. It introduces the Chain-of-Action-Thought (CoAT) framework, which models the decision-making process of GUI agents by considering the description of previous actions, the current screen state, and the reasoning behind the next actions and their expected outcomes. This approach aims to enhance the adaptability and intelligence of agents navigating complex graphical user interfaces. To support the development and evaluation of CoAT, the project provides a comprehensive benchmark dataset called Android-In-The-Zoo (AiTZ). This dataset contains 18,643 screen-action pairs annotated with CoAT-driven semantic labels, covering a wide range of instructions and scenarios. The dataset includes detailed annotations such as screen descriptions, action thoughts, action descriptions, and action results, enabling fine-grained analysis and training of GUI agents. The project also offers evaluation code and demo scripts to facilitate the use of CoAT with various proprietary models like GPT-4V, Gemini-Pro, and Qwen-VL-Max. These tools allow researchers and developers to experiment with the CoAT framework and assess its performance in real-world tasks. Statistical data about the dataset subsets, including general, install, Google apps, single, and web shopping categories, are provided to give insights into the dataset's scope and diversity. The project compares CoAT with other context modeling methods such as Chain-of-Thought (CoT) and Chain-of-Actions (CoA), demonstrating superior performance in action prediction accuracy and format hit rate. Overall, this project contributes a novel methodology and a rich dataset for advancing the capabilities of GUI agents in Android environments, supporting research in human-computer interaction, artificial intelligence, and automated software testing.
https://github.com/IMNearth/CoAT
Also in GUI Action Mapping
mobile-next/mobile-mcp
Mobile Next MCP is a platform-agnostic Model Context Protocol server enabling scalable mobile automation and interaction with iOS and Android devices, simulators, and emulators through accessibility snapshots and coordinate-based controls.
ddupont808/GPT-4V-Act
GPT-4V-Act is an AI agent that uses GPT-4V(ision) to interact with web user interfaces through mouse and keyboard inputs, enabling enhanced accessibility and automation.
naver/egjs
egjs is a modular collection of JavaScript components designed to simplify and accelerate the development of customizable web applications with a focus on ease of use and performance.
njucckevin/SeeClick
SeeClick is a state-of-the-art project providing models, data, and code for advanced visual GUI agents with a focus on GUI grounding, featuring the ScreenSpot benchmark and superior performance on multi-platform GUI element prediction.
OS-Copilot/OS-Atlas
OS-Atlas is a foundation action model for generalist GUI agents that enables precise visual grounding and interaction with UI elements in screenshots using advanced vision-language models.
AriaUI/Aria-UI
Aria-UI is an open-source, fast, and context-aware action grounding system for GUI instructions, achieving state-of-the-art performance in dynamic agent tasks like AndroidWorld and OSWorld.
opendilab/awesome-ui-agents
A curated and continuously updated collection of research papers, models, tools, and datasets focused on UI agents that interact with various user interfaces across platforms like web, mobile, and OS environments.