wandb/openui
OpenUI is an open-source tool that enables users to describe user interfaces using natural language and see them rendered live, leveraging large language models to generate UI components across multiple frontend frameworks.
Awesome AI Agents › GUI Action Mapping
zzxslp/MM-Navigator
The project "GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation" explores the use of large multimodal models (LMMs), specifically GPT-4V, to enable zero-shot navigation of smartphone graphical user interfaces (GUIs). This innovative approach leverages the capabilities of GPT-4V to understand and interact with smartphone apps without requiring task-specific training or fine-tuning. The project demonstrates the potential of LMMs to act as intelligent agents that can perform complex tasks on smartphones, such as shopping on the Amazon app, by interpreting visual and textual information from the GUI and making appropriate navigation decisions. The repository includes a demo showcasing GPT-4V navigating the Amazon app on an iPhone, highlighting the model's ability to comprehend and manipulate app interfaces in a zero-shot manner. This means the model can generalize its understanding to new apps and tasks without prior exposure, which is a significant advancement in the field of human-computer interaction and AI-driven automation. The project is grounded in research detailed in an associated paper available on arXiv, which provides the theoretical foundation and experimental validation for the approach. The authors aim to release the code and evaluation benchmark soon, which will facilitate further research and development in this area. Overall, this project represents a cutting-edge application of large multimodal models to real-world problems, showcasing how AI can enhance user interaction with technology by automating complex navigation tasks on smartphones. It is relevant for researchers and developers interested in AI, multimodal learning, human-computer interaction, and automated GUI navigation.
https://github.com/zzxslp/MM-Navigator
Also in GUI Action Mapping
mobile-next/mobile-mcp
Mobile Next MCP is a platform-agnostic Model Context Protocol server enabling scalable mobile automation and interaction with iOS and Android devices, simulators, and emulators through accessibility snapshots and coordinate-based controls.
ddupont808/GPT-4V-Act
GPT-4V-Act is an AI agent that uses GPT-4V(ision) to interact with web user interfaces through mouse and keyboard inputs, enabling enhanced accessibility and automation.
naver/egjs
egjs is a modular collection of JavaScript components designed to simplify and accelerate the development of customizable web applications with a focus on ease of use and performance.
njucckevin/SeeClick
SeeClick is a state-of-the-art project providing models, data, and code for advanced visual GUI agents with a focus on GUI grounding, featuring the ScreenSpot benchmark and superior performance on multi-platform GUI element prediction.
OS-Copilot/OS-Atlas
OS-Atlas is a foundation action model for generalist GUI agents that enables precise visual grounding and interaction with UI elements in screenshots using advanced vision-language models.
AriaUI/Aria-UI
Aria-UI is an open-source, fast, and context-aware action grounding system for GUI instructions, achieving state-of-the-art performance in dynamic agent tasks like AndroidWorld and OSWorld.
opendilab/awesome-ui-agents
A curated and continuously updated collection of research papers, models, tools, and datasets focused on UI agents that interact with various user interfaces across platforms like web, mobile, and OS environments.