wandb/openui
OpenUI is an open-source tool that enables users to describe user interfaces using natural language and see them rendered live, leveraging large language models to generate UI components across multiple frontend frameworks.
Awesome AI Agents › GUI Action Mapping
X-LANCE/Mobile-Env
Mobile-Env is a universal platform designed for training and evaluating agents that interact with mobile graphical user interfaces (GUIs). It provides a flexible and extensible environment for building qualified evaluation benchmarks for GUI interaction, particularly focusing on Android applications. The platform supports both visual and text-based observations by providing screenshots and view hierarchies, and it allows agents to perform actions such as touch inputs and typing tokens. Mobile-Env is built on top of the AndroidEnv framework and enhances it with additional features like new action and observation types, improved event management, and support for remote simulators to address hardware acceleration limitations on GPU clusters. The platform enables the creation of new tasks through task definition files, which specify crucial steps in interaction episodes, such as opening target pages or scrolling to specific areas. It supports multiple sources for parsing task events, including screen text, screen icons, view hierarchies, and system logs, making it adaptable to a wide range of real-world apps without requiring dedicated development. Mobile-Env integrates external tools like EasyOCR for optical character recognition and plans to embed icon recognition models to further enhance its capabilities. Mobile-Env also introduces advanced features such as fuzzy matching for screen text events, batch action execution to reduce network transfer delays, and various rating methods for response events, including regex, fuzzy matching, and vector encoding. The platform provides wrappers to customize observation and action spaces, supporting both continuous and discrete action spaces. It includes auxiliary tools like annotation tools and supports Docker images with pre-configured Android virtual devices for easier setup. The project is well-documented with guides on evaluating and training agents, extending environments and tasks, handling certificate pinning issues, and using auxiliary tools. It offers a demo task set based on WikiHow tasks available on the Hugging Face platform and includes examples for random and human agents. Mobile-Env is suitable for researchers and developers working on mobile UI automation, reinforcement learning, and human-computer interaction, providing a comprehensive environment for developing and benchmarking intelligent mobile agents.
https://github.com/X-LANCE/Mobile-Env
Also in GUI Action Mapping
mobile-next/mobile-mcp
Mobile Next MCP is a platform-agnostic Model Context Protocol server enabling scalable mobile automation and interaction with iOS and Android devices, simulators, and emulators through accessibility snapshots and coordinate-based controls.
ddupont808/GPT-4V-Act
GPT-4V-Act is an AI agent that uses GPT-4V(ision) to interact with web user interfaces through mouse and keyboard inputs, enabling enhanced accessibility and automation.
naver/egjs
egjs is a modular collection of JavaScript components designed to simplify and accelerate the development of customizable web applications with a focus on ease of use and performance.
njucckevin/SeeClick
SeeClick is a state-of-the-art project providing models, data, and code for advanced visual GUI agents with a focus on GUI grounding, featuring the ScreenSpot benchmark and superior performance on multi-platform GUI element prediction.
OS-Copilot/OS-Atlas
OS-Atlas is a foundation action model for generalist GUI agents that enables precise visual grounding and interaction with UI elements in screenshots using advanced vision-language models.
AriaUI/Aria-UI
Aria-UI is an open-source, fast, and context-aware action grounding system for GUI instructions, achieving state-of-the-art performance in dynamic agent tasks like AndroidWorld and OSWorld.
opendilab/awesome-ui-agents
A curated and continuously updated collection of research papers, models, tools, and datasets focused on UI agents that interact with various user interfaces across platforms like web, mobile, and OS environments.