bytedance/UI-TARS-desktop
UI-TARS Desktop is a GUI Agent application that uses a Vision-Language Model to enable natural language control of computers across multiple platforms with real-time visual recognition and automation capabilities.
Awesome AI Agents › Desktop Automation
OthersideAI/self-operating-computer
The Self-Operating Computer Framework is an innovative project designed to enable multimodal AI models to operate a computer by mimicking human inputs and outputs. The framework allows the AI to view the computer screen and decide on a series of mouse and keyboard actions to achieve specific objectives, effectively automating computer operation through AI vision and interaction. Released in November 2023, it stands as one of the pioneering examples of using multimodal models for direct computer control. The framework is compatible with various multimodal models including GPT-4o, o1, Gemini Pro Vision, Claude 3, Qwen-VL, and LLaVa, with plans to support additional models in the future. Users can install the framework via pip and run it with simple commands, providing their OpenAI or other relevant API keys for authentication. The project supports multiple modes of operation, including voice input for objectives, optical character recognition (OCR) for enhanced clickable element detection, and Set-of-Mark (SoM) prompting to improve visual grounding capabilities. The OCR mode is the default due to its superior performance in tests. The framework also integrates with local models like LLaVa through Ollama, allowing experimentation on personal machines. It supports Mac OS, Windows, and Linux platforms, requiring specific permissions for screen recording and accessibility on Mac. The project encourages community contributions and provides extensive documentation and support channels including a Discord server and social media links. It is designed to be a versatile tool for researchers and developers interested in AI-driven computer automation, leveraging the latest advancements in multimodal AI and vision-language models to create a seamless interface between AI and computer hardware.
https://github.com/OthersideAI/self-operating-computer
Also in Desktop Automation
asweigart/pyautogui
PyAutoGUI is a cross-platform Python module that enables programmatic control of the mouse and keyboard for GUI automation, including features like screen capture and image recognition.
bytebot-ai/bytebot
Bytebot is a containerized Linux desktop environment that enables programmatic control and remote access for automation, scraping, CI tasks, and remote work through a unified REST API and VNC interfaces.
wonderwhy-er/DesktopCommanderMCP
Desktop Commander MCP is a server that integrates with the Claude desktop app to provide terminal control, file system search, and diff-based file editing capabilities, enabling advanced AI-assisted desktop management.
jacklandrin/OnlySwitch
OnlySwitch is a customizable all-in-one macOS menu bar app that consolidates system toggles, shortcuts, and AI-powered controls to enhance user productivity and convenience.
iflytek/astron-rpa
AstronRPA is an enterprise-grade open-source Robotic Process Automation desktop application that enables low-code/no-code workflow automation for desktop and web applications with strong enterprise security, collaboration, and AI agent integration.
yuruotong1/autoMate
autoMate is an AI-driven local automation assistant that uses natural language commands to enable users to automate complex computer tasks without programming, supporting local deployment and full interface control for enhanced privacy and flexibility.
TurixAI/TuriX-CUA
Open-source computer-use agent that performs real GUI actions on macOS and Windows desktops, with swappable vision-language model backends and MCP access for other agents.