bytedance/UI-TARS-desktop
UI-TARS Desktop is a GUI Agent application that uses a Vision-Language Model to enable natural language control of computers across multiple platforms with real-time visual recognition and automation capabilities.
Awesome AI Agents › Desktop Automation
BAAI-Agents/Cradle
The Cradle framework is an innovative project aimed at achieving General Computer Control (GCC) by empowering foundation models to perform complex computer tasks through a unified interface that mimics human interaction with computers. It uses screenshots as input and keyboard and mouse operations as output, enabling agents to execute a wide range of tasks across different software and games. The framework supports strong reasoning abilities, self-improvement, and skill curation, making it adaptable and capable of handling diverse environments with minimal requirements. Cradle has been extended to support multiple popular games such as Red Dead Redemption 2, Stardew Valley, Cities: Skylines, and Dealer's Life 2, as well as various software applications including Chrome, Outlook, Capcut, Meitu, and Feishu. This versatility demonstrates its potential for broad applicability in both gaming and productivity software. The project provides detailed setup instructions, including environment preparation, dependency installation, and OCR tool integration, ensuring users can easily deploy and customize the framework. It also offers specific configurations and resources tailored to each supported game and software, facilitating smooth adaptation and extension to new environments. Cradle's architecture includes core modules for configuration, environment management, and skill definition, with a focus on modularity and extensibility. The framework caches models, manages prompts for different tasks, and supports automatic skill generation, enhancing its capability to learn and improve over time. Overall, Cradle represents a significant step towards creating intelligent agents capable of general computer control, combining advanced AI techniques with practical tools to interact with complex digital environments effectively. Its open-source nature and comprehensive documentation make it a valuable resource for researchers and developers interested in AI-driven automation and human-computer interaction.
https://github.com/BAAI-Agents/Cradle
Also in Desktop Automation
asweigart/pyautogui
PyAutoGUI is a cross-platform Python module that enables programmatic control of the mouse and keyboard for GUI automation, including features like screen capture and image recognition.
bytebot-ai/bytebot
Bytebot is a containerized Linux desktop environment that enables programmatic control and remote access for automation, scraping, CI tasks, and remote work through a unified REST API and VNC interfaces.
OthersideAI/self-operating-computer
The Self-Operating Computer Framework enables multimodal AI models to autonomously operate a computer by interpreting screen visuals and executing mouse and keyboard actions to achieve user-defined objectives.
wonderwhy-er/DesktopCommanderMCP
Desktop Commander MCP is a server that integrates with the Claude desktop app to provide terminal control, file system search, and diff-based file editing capabilities, enabling advanced AI-assisted desktop management.
jacklandrin/OnlySwitch
OnlySwitch is a customizable all-in-one macOS menu bar app that consolidates system toggles, shortcuts, and AI-powered controls to enhance user productivity and convenience.
iflytek/astron-rpa
AstronRPA is an enterprise-grade open-source Robotic Process Automation desktop application that enables low-code/no-code workflow automation for desktop and web applications with strong enterprise security, collaboration, and AI agent integration.
yuruotong1/autoMate
autoMate is an AI-driven local automation assistant that uses natural language commands to enable users to automate complex computer tasks without programming, supporting local deployment and full interface control for enhanced privacy and flexibility.