asweigart/pyautogui
PyAutoGUI is a cross-platform Python module that enables programmatic control of the mouse and keyboard for GUI automation, including features like screen capture and image recognition.
Awesome AI Agents › Desktop Automation
bytedance/UI-TARS-desktop
UI-TARS Desktop is a cutting-edge GUI Agent application developed by ByteDance that leverages a Vision-Language Model to enable users to control their computers using natural language commands. This innovative application integrates advanced AI capabilities to interpret visual elements on the screen and execute precise mouse and keyboard actions, providing a seamless and intuitive user experience. The project supports cross-platform functionality, working on Windows, MacOS, and even within browsers, making it highly versatile for different user environments. One of the standout features of UI-TARS Desktop is its ability to perform screenshot and visual recognition tasks, allowing it to understand and interact with graphical user interfaces dynamically. This capability is enhanced by real-time feedback and status displays, ensuring users are always informed about the agent's actions and system state. The application emphasizes privacy and security by processing all data locally, avoiding the need for cloud-based computations. The project also includes an SDK that offers a powerful toolkit for building custom GUI automation agents, expanding its utility beyond the default application. The developers have provided comprehensive documentation, including quick start guides and deployment instructions, facilitating easy adoption and integration. The UI-TARS Desktop project is actively maintained, with recent updates introducing a redesigned agent UI, improved browser operation features, and support for the advanced UI-TARS-1.5 model, which enhances performance and control precision. UI-TARS Desktop is backed by academic research, with a published paper detailing its underlying technology and methodologies. The project is open-source under the Apache License 2.0, encouraging community contributions and collaboration. Overall, UI-TARS Desktop represents a significant advancement in human-computer interaction, combining natural language processing, computer vision, and automation to create a powerful tool for controlling computers more intuitively and efficiently.
https://github.com/bytedance/UI-TARS-desktop
Also in Desktop Automation
bytebot-ai/bytebot
Bytebot is a containerized Linux desktop environment that enables programmatic control and remote access for automation, scraping, CI tasks, and remote work through a unified REST API and VNC interfaces.
OthersideAI/self-operating-computer
The Self-Operating Computer Framework enables multimodal AI models to autonomously operate a computer by interpreting screen visuals and executing mouse and keyboard actions to achieve user-defined objectives.
wonderwhy-er/DesktopCommanderMCP
Desktop Commander MCP is a server that integrates with the Claude desktop app to provide terminal control, file system search, and diff-based file editing capabilities, enabling advanced AI-assisted desktop management.
jacklandrin/OnlySwitch
OnlySwitch is a customizable all-in-one macOS menu bar app that consolidates system toggles, shortcuts, and AI-powered controls to enhance user productivity and convenience.
iflytek/astron-rpa
AstronRPA is an enterprise-grade open-source Robotic Process Automation desktop application that enables low-code/no-code workflow automation for desktop and web applications with strong enterprise security, collaboration, and AI agent integration.
yuruotong1/autoMate
autoMate is an AI-driven local automation assistant that uses natural language commands to enable users to automate complex computer tasks without programming, supporting local deployment and full interface control for enhanced privacy and flexibility.
TurixAI/TuriX-CUA
Open-source computer-use agent that performs real GUI actions on macOS and Windows desktops, with swappable vision-language model backends and MCP access for other agents.