bytedance/UI-TARS-desktop
UI-TARS Desktop is a GUI Agent application that uses a Vision-Language Model to enable natural language control of computers across multiple platforms with real-time visual recognition and automation capabilities.
Awesome AI Agents › Desktop Automation
vercel-labs/agent-desktop
agent-desktop, a Vercel Labs project, lets an AI agent operate graphical desktop applications without screenshots or pixel-based matching. It reads an application's native OS accessibility tree -- the Accessibility API on macOS, UI Automation on Windows, AT-SPI on Linux -- and exposes it to the calling agent as structured JSON, with a stable reference ("ref") for every interactive element. The agent's own loop takes a snapshot, acts on a ref (click, type, scroll, and dozens of other actions), takes a new snapshot to verify the result, and repeats; agent-desktop itself contains no model and is purely the observation/action layer. It ships as a CLI and as a C library sharing one Rust core behind a PlatformAdapter trait, with native binaries for x64 and ARM64 distributed via npm, Bun, Yarn, pnpm or Homebrew. Commands cover observation (snapshot, find, screenshot), acting on refs (click variants, type, select, toggle, scroll), raw keyboard/mouse input, window and app management, clipboard access, notifications, and session tracing -- 60 command names in total. Refs are scoped to the snapshot that produced them and become invalid once the UI changes; ambiguous targets across multiple windows must be disambiguated by window id. The tool runs headless by default, using the accessibility API without moving the cursor or stealing focus, and an explicit flag is required for physical input simulation. It also exposes every command as an MCP tool over stdio, and ships built-in, versioned skill text that an agent can read before first use. The project is Apache-2.0 licensed and written primarily in Rust.
https://github.com/vercel-labs/agent-desktop
Also in Desktop Automation
asweigart/pyautogui
PyAutoGUI is a cross-platform Python module that enables programmatic control of the mouse and keyboard for GUI automation, including features like screen capture and image recognition.
bytebot-ai/bytebot
Bytebot is a containerized Linux desktop environment that enables programmatic control and remote access for automation, scraping, CI tasks, and remote work through a unified REST API and VNC interfaces.
OthersideAI/self-operating-computer
The Self-Operating Computer Framework enables multimodal AI models to autonomously operate a computer by interpreting screen visuals and executing mouse and keyboard actions to achieve user-defined objectives.
wonderwhy-er/DesktopCommanderMCP
Desktop Commander MCP is a server that integrates with the Claude desktop app to provide terminal control, file system search, and diff-based file editing capabilities, enabling advanced AI-assisted desktop management.
jacklandrin/OnlySwitch
OnlySwitch is a customizable all-in-one macOS menu bar app that consolidates system toggles, shortcuts, and AI-powered controls to enhance user productivity and convenience.
iflytek/astron-rpa
AstronRPA is an enterprise-grade open-source Robotic Process Automation desktop application that enables low-code/no-code workflow automation for desktop and web applications with strong enterprise security, collaboration, and AI agent integration.
yuruotong1/autoMate
autoMate is an AI-driven local automation assistant that uses natural language commands to enable users to automate complex computer tasks without programming, supporting local deployment and full interface control for enhanced privacy and flexibility.