wandb/openui
OpenUI is an open-source tool that enables users to describe user interfaces using natural language and see them rendered live, leveraging large language models to generate UI components across multiple frontend frameworks.
Awesome AI Agents › GUI Action Mapping
naver/egjs
egjs is a comprehensive collection of JavaScript components designed to simplify and accelerate the process of building web applications tailored to individual needs. The project offers a modular approach, providing a variety of specialized components that developers can integrate into their projects to enhance functionality and user experience. Some of the key components included in the egjs suite are egjs-agent, egjs-axes, egjs-component, egjs-flicking, egjs-infinitegrid, egjs-persist, egjs-view360, egjs-view3d, egjs-grid, egjs-conveyer, and egjs-imready. Each component serves a distinct purpose, such as handling user input, creating interactive UI elements, managing state persistence, or rendering 3D and 360-degree views. The project is maintained by NAVER Corp and is open-source under the MIT license, ensuring it is freely available for use and modification. The official website provides detailed documentation and download links for each component, facilitating easy access and integration. Additionally, egjs supports cross-framework components, allowing developers to use these tools across different JavaScript frameworks seamlessly. The project emphasizes ease of use and speed, aiming to provide developers with the tools they need to build web applications efficiently without compromising on flexibility or performance. It also offers migration guides for users transitioning from older versions, ensuring a smooth upgrade path. The active maintenance and comprehensive documentation make egjs a valuable resource for developers looking to enhance their web applications with robust, reusable components.
Also in GUI Action Mapping
mobile-next/mobile-mcp
Mobile Next MCP is a platform-agnostic Model Context Protocol server enabling scalable mobile automation and interaction with iOS and Android devices, simulators, and emulators through accessibility snapshots and coordinate-based controls.
ddupont808/GPT-4V-Act
GPT-4V-Act is an AI agent that uses GPT-4V(ision) to interact with web user interfaces through mouse and keyboard inputs, enabling enhanced accessibility and automation.
njucckevin/SeeClick
SeeClick is a state-of-the-art project providing models, data, and code for advanced visual GUI agents with a focus on GUI grounding, featuring the ScreenSpot benchmark and superior performance on multi-platform GUI element prediction.
OS-Copilot/OS-Atlas
OS-Atlas is a foundation action model for generalist GUI agents that enables precise visual grounding and interaction with UI elements in screenshots using advanced vision-language models.
AriaUI/Aria-UI
Aria-UI is an open-source, fast, and context-aware action grounding system for GUI instructions, achieving state-of-the-art performance in dynamic agent tasks like AndroidWorld and OSWorld.
opendilab/awesome-ui-agents
A curated and continuously updated collection of research papers, models, tools, and datasets focused on UI agents that interact with various user interfaces across platforms like web, mobile, and OS environments.
cooelf/Auto-GUI
Auto-GUI is a multimodal AI agent framework that predicts user interface actions using a novel chain-of-action technique, enabling direct interaction with interfaces without environment parsing or application-specific APIs.