wandb/openui
OpenUI is an open-source tool that enables users to describe user interfaces using natural language and see them rendered live, leveraging large language models to generate UI components across multiple frontend frameworks.
Awesome AI Agents › GUI Action Mapping
mobile-next/mobile-mcp
Mobile Next - MCP server for Mobile Development and Automation is a Model Context Protocol (MCP) server designed to enable scalable mobile automation and development through a platform-agnostic interface. It supports iOS, Android, simulators, emulators, and physical devices, allowing interaction with native mobile applications without requiring distinct knowledge of iOS or Android platforms. The server facilitates communication between agents and large language models (LLMs) with mobile devices using structured accessibility snapshots or coordinate-based taps derived from screenshots. This approach supports native app automation for testing, data entry, scripted flows, multi-step user journeys driven by LLMs, and general-purpose mobile application interaction for agent-based frameworks. It also enables agent-to-agent communication for mobile automation and data extraction use cases. Key features include fast and lightweight operation using native accessibility trees or screenshot-based coordinates when accessibility labels are unavailable. It is LLM-friendly, requiring no computer vision models for accessibility snapshots, and incorporates visual sense capabilities to analyze screen content and decide actions. The server reduces ambiguity by relying on structured data whenever possible and supports extracting structured data from visible screen elements. The architecture supports connection to iOS simulators on macOS/Linux, Android emulators on multiple platforms, and physical devices with proper tools and drivers. It can run in headless mode on simulators and emulators, enabling background operation without physical devices. The server provides a comprehensive set of mobile commands and interaction tools, including listing installed apps, launching and terminating apps, getting screen size, tapping on screen coordinates or UI elements, pressing device buttons, opening URLs, typing text, and performing swipe gestures. Installation requires Xcode command line tools, Android platform tools, Node.js, and MCP-supported foundational models or agents. The project includes detailed documentation, a wiki for setup and debugging, and a roadmap for future enhancements. Mobile Next MCP aims to simplify and scale mobile automation by providing a unified, extensible platform for interacting with mobile devices and applications across different environments.
https://github.com/mobile-next/mobile-mcp
Also in GUI Action Mapping
ddupont808/GPT-4V-Act
GPT-4V-Act is an AI agent that uses GPT-4V(ision) to interact with web user interfaces through mouse and keyboard inputs, enabling enhanced accessibility and automation.
naver/egjs
egjs is a modular collection of JavaScript components designed to simplify and accelerate the development of customizable web applications with a focus on ease of use and performance.
njucckevin/SeeClick
SeeClick is a state-of-the-art project providing models, data, and code for advanced visual GUI agents with a focus on GUI grounding, featuring the ScreenSpot benchmark and superior performance on multi-platform GUI element prediction.
OS-Copilot/OS-Atlas
OS-Atlas is a foundation action model for generalist GUI agents that enables precise visual grounding and interaction with UI elements in screenshots using advanced vision-language models.
AriaUI/Aria-UI
Aria-UI is an open-source, fast, and context-aware action grounding system for GUI instructions, achieving state-of-the-art performance in dynamic agent tasks like AndroidWorld and OSWorld.
opendilab/awesome-ui-agents
A curated and continuously updated collection of research papers, models, tools, and datasets focused on UI agents that interact with various user interfaces across platforms like web, mobile, and OS environments.
cooelf/Auto-GUI
Auto-GUI is a multimodal AI agent framework that predicts user interface actions using a novel chain-of-action technique, enabling direct interaction with interfaces without environment parsing or application-specific APIs.