bytedance/UI-TARS-desktop
UI-TARS Desktop is a GUI Agent application that uses a Vision-Language Model to enable natural language control of computers across multiple platforms with real-time visual recognition and automation capabilities.
Awesome AI Agents › Desktop Automation
showlab/WorldGUI
WorldGUI is an advanced interactive benchmark and agent framework designed for desktop GUI automation, enabling AI to control a PC from any starting point. The project introduces the WorldGUI-Agent, a highly adaptive GUI agent that leverages a novel self-reflection mechanism to perform tasks in dynamic graphical user interface environments. This agent incorporates three key self-reflection modules: Planner-Critic for post-planning critique to refine initial plans, Step-Check for pre-execution validation to remove or modify redundant steps, and Actor-Critic for post-action evaluation to assess task completion and apply corrections. The framework supports both instructional video inputs and non-video inputs, allowing flexible interaction modes for task execution. It integrates with a wide range of large language models (LMMs) such as GPT-4o, Gemini-2.0, Claude-3.5-Sonnet, and Claude-3.7-Sonnet, providing universal LMM support. The system includes a GUI Parser that uses Google OCR and PyAutoGUI to extract element information from the desktop environment, a State-Aware Planner that generates plans based on screenshots and videos, and an Actor module that translates action descriptions into executable commands. The project emphasizes easy deployment without the need for Docker or virtual machines, enabling users to start quickly with simple commands. WorldGUI outperforms existing state-of-the-art desktop GUI agents, showing significant improvements in benchmark performance. The repository also offers a frontend-backend communication system for flexible deployment and supports locally running models and user interfaces. The project is actively maintained with ongoing improvements, including a fast version of the agent and plans for user-friendly interfaces and online demos. Overall, WorldGUI represents a cutting-edge solution for automating desktop GUI tasks through AI, combining robust planning, validation, and execution capabilities in a comprehensive framework.
https://github.com/showlab/WorldGUI
Also in Desktop Automation
asweigart/pyautogui
PyAutoGUI is a cross-platform Python module that enables programmatic control of the mouse and keyboard for GUI automation, including features like screen capture and image recognition.
bytebot-ai/bytebot
Bytebot is a containerized Linux desktop environment that enables programmatic control and remote access for automation, scraping, CI tasks, and remote work through a unified REST API and VNC interfaces.
OthersideAI/self-operating-computer
The Self-Operating Computer Framework enables multimodal AI models to autonomously operate a computer by interpreting screen visuals and executing mouse and keyboard actions to achieve user-defined objectives.
wonderwhy-er/DesktopCommanderMCP
Desktop Commander MCP is a server that integrates with the Claude desktop app to provide terminal control, file system search, and diff-based file editing capabilities, enabling advanced AI-assisted desktop management.
jacklandrin/OnlySwitch
OnlySwitch is a customizable all-in-one macOS menu bar app that consolidates system toggles, shortcuts, and AI-powered controls to enhance user productivity and convenience.
iflytek/astron-rpa
AstronRPA is an enterprise-grade open-source Robotic Process Automation desktop application that enables low-code/no-code workflow automation for desktop and web applications with strong enterprise security, collaboration, and AI agent integration.
yuruotong1/autoMate
autoMate is an AI-driven local automation assistant that uses natural language commands to enable users to automate complex computer tasks without programming, supporting local deployment and full interface control for enhanced privacy and flexibility.