bytedance/UI-TARS-desktop
UI-TARS Desktop is a GUI Agent application that uses a Vision-Language Model to enable natural language control of computers across multiple platforms with real-time visual recognition and automation capabilities.
Awesome AI Agents › Desktop Automation
instavm/clickclickclick
ClickClickClick is a computer-use framework that lets a language model operate an Android phone or a macOS desktop from a plain-language instruction. The architecture splits the work between two models: a planner that decides the next step of the task and a finder that inspects a screenshot to locate the relevant user-interface element, after which the framework issues the actual tap, click or keystroke. Either role can be filled by a different backend, so a strong remote model can plan while a cheaper or local model handles element detection, or the reverse. Supported providers are OpenAI, Google Gemini and locally hosted models through Ollama, including vision-capable options such as Llama 3.2-vision and qwen3.5:4b; the maintainers report the best current results using a Gemini Flash-Lite model in both roles and note that small local models can plan simple navigation but detect UI elements unreliably. An --image-quality flag scales screenshots down to trade accuracy for latency and token cost, which is recommended when running locally. The project offers four entry points: a Gradio web interface, a click3 command-line tool whose run subcommand takes the task prompt plus --platform, --planner-model and --finder-model options, a Python API for embedding the loop in other code, and a REST API for remote invocation. A setup command walks through choosing models and supplying API keys, and per-model settings live in config/models.yaml with keys read from the environment. Demonstrations in the repository show it drafting a Gmail message, answering a question by navigating Google Maps and starting a game on lichess. The authors describe the code as highly experimental and expected to change, and advise using it at your own risk. It is written in Python and targets developers exploring device automation with vision language models.
https://github.com/instavm/clickclickclick
Also in Desktop Automation
asweigart/pyautogui
PyAutoGUI is a cross-platform Python module that enables programmatic control of the mouse and keyboard for GUI automation, including features like screen capture and image recognition.
bytebot-ai/bytebot
Bytebot is a containerized Linux desktop environment that enables programmatic control and remote access for automation, scraping, CI tasks, and remote work through a unified REST API and VNC interfaces.
OthersideAI/self-operating-computer
The Self-Operating Computer Framework enables multimodal AI models to autonomously operate a computer by interpreting screen visuals and executing mouse and keyboard actions to achieve user-defined objectives.
wonderwhy-er/DesktopCommanderMCP
Desktop Commander MCP is a server that integrates with the Claude desktop app to provide terminal control, file system search, and diff-based file editing capabilities, enabling advanced AI-assisted desktop management.
jacklandrin/OnlySwitch
OnlySwitch is a customizable all-in-one macOS menu bar app that consolidates system toggles, shortcuts, and AI-powered controls to enhance user productivity and convenience.
iflytek/astron-rpa
AstronRPA is an enterprise-grade open-source Robotic Process Automation desktop application that enables low-code/no-code workflow automation for desktop and web applications with strong enterprise security, collaboration, and AI agent integration.
yuruotong1/autoMate
autoMate is an AI-driven local automation assistant that uses natural language commands to enable users to automate complex computer tasks without programming, supporting local deployment and full interface control for enhanced privacy and flexibility.