wandb/openui
OpenUI is an open-source tool that enables users to describe user interfaces using natural language and see them rendered live, leveraging large language models to generate UI components across multiple frontend frameworks.
Awesome AI Agents › GUI Action Mapping
AriaUI/Aria-UI
Aria-UI is an open-source project focused on fast and context-aware action grounding from GUI instructions, designed for GUI and computer-use agents. It excels in interpreting diverse grounding instructions, including text and text-image interleaved formats, to improve the accuracy of grounding in dynamic scenarios. The project features a lightweight and efficient mixture-of-expert model with 3.9 billion activated parameters per token, capable of encoding GUI inputs of varying sizes and aspect ratios with ultra-resolution support. Aria-UI achieves state-of-the-art performance on offline and online agent benchmarks, securing first place on AndroidWorld with a 44.8% task success rate and third place on OSWorld with a 15.2% task success rate as of December 2024. The project provides context-aware grounding by leveraging historical input to enhance grounding accuracy. It supports versatile grounding instruction understanding, making it adaptable to various planning agents and dynamic environments. Aria-UI offers models, datasets, and context-aware datasets available on Hugging Face, along with a demo and a project page for easy access and experimentation. Installation instructions include dependencies such as transformers, accelerate, sentencepiece, torchvision, requests, torch, Pillow, and flash-attn, with an optional grouped-gemm for improved inference performance. The project recommends using the vLLM library for inference, providing code snippets for integration. Alternatively, it supports inference using the transformers API. Aria-UI is backed by academic research, with a cited paper available on arXiv, and acknowledges contributors for valuable discussions and suggestions. The project aims to enhance task success rates and seamless integration in GUI-based agent tasks, particularly in environments like AndroidWorld and OSWorld, making it a significant tool for advancing GUI instruction grounding and agent interaction capabilities.
https://github.com/AriaUI/Aria-UI
Also in GUI Action Mapping
mobile-next/mobile-mcp
Mobile Next MCP is a platform-agnostic Model Context Protocol server enabling scalable mobile automation and interaction with iOS and Android devices, simulators, and emulators through accessibility snapshots and coordinate-based controls.
ddupont808/GPT-4V-Act
GPT-4V-Act is an AI agent that uses GPT-4V(ision) to interact with web user interfaces through mouse and keyboard inputs, enabling enhanced accessibility and automation.
naver/egjs
egjs is a modular collection of JavaScript components designed to simplify and accelerate the development of customizable web applications with a focus on ease of use and performance.
njucckevin/SeeClick
SeeClick is a state-of-the-art project providing models, data, and code for advanced visual GUI agents with a focus on GUI grounding, featuring the ScreenSpot benchmark and superior performance on multi-platform GUI element prediction.
OS-Copilot/OS-Atlas
OS-Atlas is a foundation action model for generalist GUI agents that enables precise visual grounding and interaction with UI elements in screenshots using advanced vision-language models.
opendilab/awesome-ui-agents
A curated and continuously updated collection of research papers, models, tools, and datasets focused on UI agents that interact with various user interfaces across platforms like web, mobile, and OS environments.
cooelf/Auto-GUI
Auto-GUI is a multimodal AI agent framework that predicts user interface actions using a novel chain-of-action technique, enabling direct interaction with interfaces without environment parsing or application-specific APIs.