bytedance/UI-TARS-desktop
UI-TARS Desktop is a GUI Agent application that uses a Vision-Language Model to enable natural language control of computers across multiple platforms with real-time visual recognition and automation capabilities.
Awesome AI Agents › Desktop Automation
e2b-dev/open-computer-use
Open Computer Use is a secure cloud Linux computer environment powered by the E2B Desktop Sandbox and controlled by open-source large language models (LLMs). It enables users to operate a virtual computer through keyboard, mouse, and shell commands, with the display live-streamed to the client computer. The system supports over 10 different LLMs, including OS-Atlas, ShowUI, and many others, allowing for flexible integration of various AI models to control the computer. The project is designed primarily for Ubuntu but is adaptable to other operating systems as well. The architecture of Open Computer Use is centered around providing a secure and interactive AI-driven computing experience. The sandbox environment ensures security while the LLMs provide the intelligence to operate the computer autonomously or semi-autonomously. Users can pause the AI agent and provide prompts at any time, making the interaction dynamic and user-controlled. The project supports a wide range of LLM providers such as Fireworks, OpenRouter, Groq, DeepSeek, Google Gemini, OpenAI GPT-4o, Anthropic Claude, HuggingFace Spaces, Moonshot, and Mistral AI. This extensive support allows users to select or swap models based on their needs. The configuration of these models is managed through a simple Python configuration file, making it easy to customize the AI's capabilities. To get started, users need Python 3.10 or later, git, an E2B API key, and API keys for the chosen LLM providers. The setup involves installing dependencies, cloning the repository, setting environment variables for API keys, and running the web interface. The project is well-documented with clear instructions and encourages contributions for adding new models or providers. Overall, Open Computer Use is an innovative project that combines cloud computing, sandbox security, and advanced AI models to create a versatile and secure AI-operated computer environment.
https://github.com/e2b-dev/open-computer-use
Also in Desktop Automation
asweigart/pyautogui
PyAutoGUI is a cross-platform Python module that enables programmatic control of the mouse and keyboard for GUI automation, including features like screen capture and image recognition.
bytebot-ai/bytebot
Bytebot is a containerized Linux desktop environment that enables programmatic control and remote access for automation, scraping, CI tasks, and remote work through a unified REST API and VNC interfaces.
OthersideAI/self-operating-computer
The Self-Operating Computer Framework enables multimodal AI models to autonomously operate a computer by interpreting screen visuals and executing mouse and keyboard actions to achieve user-defined objectives.
wonderwhy-er/DesktopCommanderMCP
Desktop Commander MCP is a server that integrates with the Claude desktop app to provide terminal control, file system search, and diff-based file editing capabilities, enabling advanced AI-assisted desktop management.
jacklandrin/OnlySwitch
OnlySwitch is a customizable all-in-one macOS menu bar app that consolidates system toggles, shortcuts, and AI-powered controls to enhance user productivity and convenience.
iflytek/astron-rpa
AstronRPA is an enterprise-grade open-source Robotic Process Automation desktop application that enables low-code/no-code workflow automation for desktop and web applications with strong enterprise security, collaboration, and AI agent integration.
yuruotong1/autoMate
autoMate is an AI-driven local automation assistant that uses natural language commands to enable users to automate complex computer tasks without programming, supporting local deployment and full interface control for enhanced privacy and flexibility.