wandb/openui
OpenUI is an open-source tool that enables users to describe user interfaces using natural language and see them rendered live, leveraging large language models to generate UI components across multiple frontend frameworks.
Awesome AI Agents › GUI Action Mapping
ddupont808/GPT-4V-Act
GPT-4V-Act is an advanced AI agent that integrates GPT-4V(ision) with a web browser to enable human-like interaction with web user interfaces through mouse and keyboard inputs. The project aims to bridge the gap between human-computer interaction by providing a seamless interface where the AI can perceive screen feedback and perform low-level input actions, effectively acting as a digital assistant for UI navigation and control. This capability is particularly useful for enhancing accessibility, automating workflows, and conducting automated UI testing. The core technology behind GPT-4V-Act combines GPT-4V(ision) with a novel approach called Set-of-Mark Prompting, supported by a custom auto-labeler that assigns unique numerical IDs to interactable UI elements. This labeling system allows the AI to precisely identify and interact with specific parts of the interface by referencing these IDs for mouse clicks or keyboard typing actions. The system processes a task description along with a screenshot of the UI to determine the next appropriate action, such as clicking a button or typing text. The project currently supports partial vision capabilities, including a JavaScript DOM auto-labeler with COCO export functionality, and basic interaction features like clicking and typing alphanumeric characters. However, some features like AI auto-labeling, special keycode typing, scrolling, user prompting for additional information, and memory of task-relevant information are still under development or not implemented. GPT-4V-Act is designed to be easily set up and demonstrated, with clear instructions for cloning the repository, installing dependencies, and running a demo. The project also encourages community involvement for feedback and contributions. A notable use case demonstrated is instructing the agent to perform tasks such as playing a random song by interpreting user prompts and interacting with the web UI accordingly. Overall, GPT-4V-Act represents a significant step towards creating intelligent, multimodal AI assistants capable of sophisticated interaction with digital environments, enhancing both accessibility and automation potential.
https://github.com/ddupont808/GPT-4V-Act
Also in GUI Action Mapping
mobile-next/mobile-mcp
Mobile Next MCP is a platform-agnostic Model Context Protocol server enabling scalable mobile automation and interaction with iOS and Android devices, simulators, and emulators through accessibility snapshots and coordinate-based controls.
naver/egjs
egjs is a modular collection of JavaScript components designed to simplify and accelerate the development of customizable web applications with a focus on ease of use and performance.
njucckevin/SeeClick
SeeClick is a state-of-the-art project providing models, data, and code for advanced visual GUI agents with a focus on GUI grounding, featuring the ScreenSpot benchmark and superior performance on multi-platform GUI element prediction.
OS-Copilot/OS-Atlas
OS-Atlas is a foundation action model for generalist GUI agents that enables precise visual grounding and interaction with UI elements in screenshots using advanced vision-language models.
AriaUI/Aria-UI
Aria-UI is an open-source, fast, and context-aware action grounding system for GUI instructions, achieving state-of-the-art performance in dynamic agent tasks like AndroidWorld and OSWorld.
opendilab/awesome-ui-agents
A curated and continuously updated collection of research papers, models, tools, and datasets focused on UI agents that interact with various user interfaces across platforms like web, mobile, and OS environments.
cooelf/Auto-GUI
Auto-GUI is a multimodal AI agent framework that predicts user interface actions using a novel chain-of-action technique, enabling direct interaction with interfaces without environment parsing or application-specific APIs.