wandb/openui
OpenUI is an open-source tool that enables users to describe user interfaces using natural language and see them rendered live, leveraging large language models to generate UI components across multiple frontend frameworks.
Awesome AI Agents › GUI Action Mapping
cooelf/Auto-GUI
Auto-GUI is the official implementation of the research project "You Only Look at Screens: Multimodal Chain-of-Action Agents," presented in the Findings of ACL 2024. This project introduces a novel multimodal solution designed to interact directly with user interfaces without relying on environment parsing or application-specific APIs. The core innovation lies in the chain-of-action technique, which enhances the agent's ability to predict actions by considering a sequence of past actions and planned future actions. This approach allows the system to better understand and anticipate user interactions with graphical interfaces. The project provides a comprehensive framework for training and inference of these multimodal agents. It includes scripts for downloading and processing datasets, specifically the AITW dataset, which is essential for training the models. The dataset is organized with various splits and feature extraction options to facilitate effective training. The training process leverages distributed computing with multiple GPUs and uses a transformer-based model architecture, specifically the "declare-lab/flan-alpaca-base" model, to handle the multimodal input data. Auto-GUI supports detailed configuration options for training, such as learning rate, batch size, input and output sequence lengths, and the use of historical and future action data. The inference process is similarly well-documented, allowing users to apply trained models to new data for action prediction tasks. The project is well-suited for researchers and developers interested in human-computer interaction, multimodal machine learning, and AI agents capable of understanding and interacting with complex user interfaces. It is open-source under the Apache-2.0 License, encouraging further research and development in this innovative area of AI-driven interface interaction.
https://github.com/cooelf/Auto-GUI
Also in GUI Action Mapping
mobile-next/mobile-mcp
Mobile Next MCP is a platform-agnostic Model Context Protocol server enabling scalable mobile automation and interaction with iOS and Android devices, simulators, and emulators through accessibility snapshots and coordinate-based controls.
ddupont808/GPT-4V-Act
GPT-4V-Act is an AI agent that uses GPT-4V(ision) to interact with web user interfaces through mouse and keyboard inputs, enabling enhanced accessibility and automation.
naver/egjs
egjs is a modular collection of JavaScript components designed to simplify and accelerate the development of customizable web applications with a focus on ease of use and performance.
njucckevin/SeeClick
SeeClick is a state-of-the-art project providing models, data, and code for advanced visual GUI agents with a focus on GUI grounding, featuring the ScreenSpot benchmark and superior performance on multi-platform GUI element prediction.
OS-Copilot/OS-Atlas
OS-Atlas is a foundation action model for generalist GUI agents that enables precise visual grounding and interaction with UI elements in screenshots using advanced vision-language models.
AriaUI/Aria-UI
Aria-UI is an open-source, fast, and context-aware action grounding system for GUI instructions, achieving state-of-the-art performance in dynamic agent tasks like AndroidWorld and OSWorld.
opendilab/awesome-ui-agents
A curated and continuously updated collection of research papers, models, tools, and datasets focused on UI agents that interact with various user interfaces across platforms like web, mobile, and OS environments.