bytedance/UI-TARS-desktop
UI-TARS Desktop is a GUI Agent application that uses a Vision-Language Model to enable natural language control of computers across multiple platforms with real-time visual recognition and automation capabilities.
Awesome AI Agents › Desktop Automation
showlab/computer_use_ootb
Computer Use OOTB is an out-of-the-box graphical user interface (GUI) agent designed for Windows and macOS platforms. It provides a ready-to-use solution for desktop GUI automation and interaction, integrating API-based models like Claude 3.5 Computer Use and locally running models such as ShowUI and UI-TARS. The project emphasizes ease of use by eliminating the need for Docker and offering a user-friendly interface built on Gradio. It supports advanced features like multi-display usage, high-resolution displays with low screenshot token costs, and remote control of computers via mobile devices without requiring mobile app installation. The system supports local running of ShowUI, a 2-billion parameter vision-language-action model, which offers a cost-effective alternative to cloud-based solutions, significantly reducing operational costs. The project is actively maintained with frequent updates, including support for UI-TARS deployment both locally and on the cloud, and enhancements to the Claude 3.5 Computer Use model. It also provides detailed setup instructions, including hardware requirements for optimal performance, dependency installation, and environment variable configuration for API keys. The project is well-documented with a project page, demo videos, and a research paper available on arXiv, highlighting its academic and practical contributions to GUI agent technology. Overall, Computer Use OOTB aims to simplify and democratize the use of AI-powered GUI agents for desktop environments, making it accessible for users to control and automate their computers efficiently through natural language commands and visual interaction.
https://github.com/showlab/computer_use_ootb
Also in Desktop Automation
asweigart/pyautogui
PyAutoGUI is a cross-platform Python module that enables programmatic control of the mouse and keyboard for GUI automation, including features like screen capture and image recognition.
bytebot-ai/bytebot
Bytebot is a containerized Linux desktop environment that enables programmatic control and remote access for automation, scraping, CI tasks, and remote work through a unified REST API and VNC interfaces.
OthersideAI/self-operating-computer
The Self-Operating Computer Framework enables multimodal AI models to autonomously operate a computer by interpreting screen visuals and executing mouse and keyboard actions to achieve user-defined objectives.
wonderwhy-er/DesktopCommanderMCP
Desktop Commander MCP is a server that integrates with the Claude desktop app to provide terminal control, file system search, and diff-based file editing capabilities, enabling advanced AI-assisted desktop management.
jacklandrin/OnlySwitch
OnlySwitch is a customizable all-in-one macOS menu bar app that consolidates system toggles, shortcuts, and AI-powered controls to enhance user productivity and convenience.
iflytek/astron-rpa
AstronRPA is an enterprise-grade open-source Robotic Process Automation desktop application that enables low-code/no-code workflow automation for desktop and web applications with strong enterprise security, collaboration, and AI agent integration.
yuruotong1/autoMate
autoMate is an AI-driven local automation assistant that uses natural language commands to enable users to automate complex computer tasks without programming, supporting local deployment and full interface control for enhanced privacy and flexibility.