wandb/openui
OpenUI is an open-source tool that enables users to describe user interfaces using natural language and see them rendered live, leveraging large language models to generate UI components across multiple frontend frameworks.
Awesome AI Agents › GUI Action Mapping
OS-Copilot/OS-Atlas
OS-Atlas is a foundation action model designed for generalist GUI agents, aiming to enhance interaction with graphical user interfaces through advanced visual grounding capabilities. The project provides two base grounding models, OS-Atlas-Base-4B and OS-Atlas-Base-7B, which are fine-tuned from existing vision-language models to accurately identify and locate UI elements within screenshots. These models accept images of any size and output normalized coordinates for UI elements, facilitating precise interaction such as clicking or switching language on a webpage. The repository includes detailed instructions for inference using both models, with code examples demonstrating how to preprocess images, load models, and generate responses that specify the coordinates of UI elements based on user queries. OS-Atlas supports multi-GPU setups and integrates with popular libraries like transformers and qwen-vl-utils for efficient model loading and processing. The project is academically recognized, with its paper accepted at ICLR 2025, and provides resources such as datasets and pre-trained models on Hugging Face. OS-Atlas aims to serve as a foundational tool for developing intelligent agents capable of understanding and manipulating GUIs across diverse applications, enhancing automation and user interaction through AI-driven visual grounding and action modeling.
https://github.com/OS-Copilot/OS-Atlas
Also in GUI Action Mapping
mobile-next/mobile-mcp
Mobile Next MCP is a platform-agnostic Model Context Protocol server enabling scalable mobile automation and interaction with iOS and Android devices, simulators, and emulators through accessibility snapshots and coordinate-based controls.
ddupont808/GPT-4V-Act
GPT-4V-Act is an AI agent that uses GPT-4V(ision) to interact with web user interfaces through mouse and keyboard inputs, enabling enhanced accessibility and automation.
naver/egjs
egjs is a modular collection of JavaScript components designed to simplify and accelerate the development of customizable web applications with a focus on ease of use and performance.
njucckevin/SeeClick
SeeClick is a state-of-the-art project providing models, data, and code for advanced visual GUI agents with a focus on GUI grounding, featuring the ScreenSpot benchmark and superior performance on multi-platform GUI element prediction.
AriaUI/Aria-UI
Aria-UI is an open-source, fast, and context-aware action grounding system for GUI instructions, achieving state-of-the-art performance in dynamic agent tasks like AndroidWorld and OSWorld.
opendilab/awesome-ui-agents
A curated and continuously updated collection of research papers, models, tools, and datasets focused on UI agents that interact with various user interfaces across platforms like web, mobile, and OS environments.
cooelf/Auto-GUI
Auto-GUI is a multimodal AI agent framework that predicts user interface actions using a novel chain-of-action technique, enabling direct interaction with interfaces without environment parsing or application-specific APIs.