Awesome AI Agents › GUI Action Mapping
OpenUI is an innovative open-source project designed to simplify and accelerate the process of building user interface (UI) components by leveraging the power of large language models (LLMs). The core idea behind OpenUI is to allow users to describe their UI concepts using natural language or imagination, and then see these descriptions rendered live into actual UI components. This approach transforms the traditionally tedious and complex task of UI development into a more enjoyable, fast, and flexible experience. The project supports integration with multiple LLM providers including OpenAI, Groq, Gemini, Anthropic, Cohere, Mistral, and others through the LiteLLM interface, enabling users to choose from a variety of AI models for generating UI code. OpenUI can convert described UI elements into various frontend frameworks and technologies such as React, Svelte, and Web Components, making it versatile for different development environments. OpenUI is also designed with ease of use in mind, offering a live demo for immediate experimentation and detailed instructions for running the tool locally using Docker, Python, or development containers. It supports advanced configurations like custom LiteLLM proxy settings and integration with Ollama models for local AI inference. The project is actively used by the Weights & Biases (W&B) team to prototype next-generation tooling for building powerful applications on top of LLMs. The repository includes comprehensive documentation and setup guides for different environments including Docker Compose, Codespaces, and Gitpod, facilitating quick onboarding for developers. OpenUI's architecture separates frontend and backend components, each with its own README for further details. The project aims to democratize UI development by making it accessible through natural language descriptions and AI-powered code generation, fostering creativity and productivity in UI design and development.
https://github.com/wandb/openui
Mobile Next MCP is a platform-agnostic Model Context Protocol server enabling scalable mobile automation and interaction with iOS and Android devices, simulators, and emulators through accessibility snapshots and coordinate-based controls.
GPT-4V-Act is an AI agent that uses GPT-4V(ision) to interact with web user interfaces through mouse and keyboard inputs, enabling enhanced accessibility and automation.
egjs is a modular collection of JavaScript components designed to simplify and accelerate the development of customizable web applications with a focus on ease of use and performance.
SeeClick is a state-of-the-art project providing models, data, and code for advanced visual GUI agents with a focus on GUI grounding, featuring the ScreenSpot benchmark and superior performance on multi-platform GUI element prediction.
OS-Atlas is a foundation action model for generalist GUI agents that enables precise visual grounding and interaction with UI elements in screenshots using advanced vision-language models.
Aria-UI is an open-source, fast, and context-aware action grounding system for GUI instructions, achieving state-of-the-art performance in dynamic agent tasks like AndroidWorld and OSWorld.
A curated and continuously updated collection of research papers, models, tools, and datasets focused on UI agents that interact with various user interfaces across platforms like web, mobile, and OS environments.
Auto-GUI is a multimodal AI agent framework that predicts user interface actions using a novel chain-of-action technique, enabling direct interaction with interfaces without environment parsing or application-specific APIs.