bytedance/UI-TARS-desktop
UI-TARS Desktop is a GUI Agent application that uses a Vision-Language Model to enable natural language control of computers across multiple platforms with real-time visual recognition and automation capabilities.
Awesome AI Agents › Desktop Automation
CursorTouch/Windows-Use
Windows-Use is an open-source automation agent designed specifically for the Windows operating system, supporting versions from Windows 7 through Windows 11. It enables AI agents, particularly large language models (LLMs), to interact directly with the Windows graphical user interface (GUI) layer to perform a variety of tasks. These tasks include opening applications, clicking buttons, typing text, executing shell commands, and capturing the state of the user interface. Unlike traditional automation tools that rely heavily on computer vision models, Windows-Use operates without such dependencies, making it a versatile and efficient bridge between AI and Windows OS automation. The project is built to work with Python 3.12 or higher and can be installed easily via pip or the UV package manager. It supports integration with popular LLMs such as Ollama and Google Gemini, allowing users to create intelligent agents that can understand and execute commands on a Windows machine. The agent can be configured to use or bypass vision capabilities and can automatically minimize windows during operation for a smoother user experience. Windows-Use is particularly useful for developers and researchers looking to automate complex workflows on Windows using AI-driven commands. It provides a safe and intelligent interface for automation but advises users to run the agent in sandbox environments to prevent unintended system changes. The project also includes demos showcasing its capabilities, such as writing notes and changing system themes, demonstrating practical applications of AI-powered Windows automation. The project is licensed under the MIT License and acknowledges the use of other open-source libraries like UIAutomation and PyAutoGUI, which contribute to its functionality. Contributions to the project are welcomed, and the repository provides detailed setup and development guidelines. Overall, Windows-Use represents a significant step forward in enabling AI to control and automate Windows OS tasks seamlessly and intelligently.
https://github.com/CursorTouch/Windows-Use
Also in Desktop Automation
asweigart/pyautogui
PyAutoGUI is a cross-platform Python module that enables programmatic control of the mouse and keyboard for GUI automation, including features like screen capture and image recognition.
bytebot-ai/bytebot
Bytebot is a containerized Linux desktop environment that enables programmatic control and remote access for automation, scraping, CI tasks, and remote work through a unified REST API and VNC interfaces.
OthersideAI/self-operating-computer
The Self-Operating Computer Framework enables multimodal AI models to autonomously operate a computer by interpreting screen visuals and executing mouse and keyboard actions to achieve user-defined objectives.
wonderwhy-er/DesktopCommanderMCP
Desktop Commander MCP is a server that integrates with the Claude desktop app to provide terminal control, file system search, and diff-based file editing capabilities, enabling advanced AI-assisted desktop management.
jacklandrin/OnlySwitch
OnlySwitch is a customizable all-in-one macOS menu bar app that consolidates system toggles, shortcuts, and AI-powered controls to enhance user productivity and convenience.
iflytek/astron-rpa
AstronRPA is an enterprise-grade open-source Robotic Process Automation desktop application that enables low-code/no-code workflow automation for desktop and web applications with strong enterprise security, collaboration, and AI agent integration.
yuruotong1/autoMate
autoMate is an AI-driven local automation assistant that uses natural language commands to enable users to automate complex computer tasks without programming, supporting local deployment and full interface control for enhanced privacy and flexibility.