bytedance/UI-TARS-desktop
UI-TARS Desktop is a GUI Agent application that uses a Vision-Language Model to enable natural language control of computers across multiple platforms with real-time visual recognition and automation capabilities.
Awesome AI Agents › Desktop Automation
askui/vision-agent
AskUI Vision Agent is a comprehensive automation framework designed to enable secure and efficient automation of native devices across multiple operating systems including Windows, Linux, MacOS, Android, and iOS. It supports Citrix environments and offers both single-step UI automation commands similar to Robotic Process Automation (RPA) and more advanced agentic intent-based instructions. One of its standout features is the ability to perform in-background automation on Windows machines, allowing the agent to operate in a separate session without interfering with the user's active session. The system supports flexible model usage, including hot swapping of AI models and infrastructure for retraining models, which can be deployed on-premise for enterprise security. The project provides an Agent OS that acts as a device controller capable of taking screenshots, moving the mouse, clicking, and typing on the keyboard across any supported operating system. Installation packages are available for different architectures and platforms, with specific instructions for Windows, Linux (not supporting Wayland currently), and MacOS. The Vision Agent Python package can be installed via pip and requires Python 3.10 or higher. Authentication with AI model providers is necessary to leverage the AI capabilities of the Vision Agent. Supported providers include AskUI's own model service and Anthropic, each with different supported commands and features. Additionally, users can test the agent with Hugging Face AI models via their Spaces API or host their own AI models such as UI-TARS by providing the appropriate API endpoints. The Vision Agent allows users to automate tasks programmatically using Python, with commands to click, type, and extract information from the screen, as well as perform autonomous actions. It supports specifying different AI models for different commands to optimize performance and accuracy. The project is enterprise-focused, emphasizing secure deployment and flexible, scalable automation solutions for native device control.
https://github.com/askui/vision-agent
Also in Desktop Automation
asweigart/pyautogui
PyAutoGUI is a cross-platform Python module that enables programmatic control of the mouse and keyboard for GUI automation, including features like screen capture and image recognition.
bytebot-ai/bytebot
Bytebot is a containerized Linux desktop environment that enables programmatic control and remote access for automation, scraping, CI tasks, and remote work through a unified REST API and VNC interfaces.
OthersideAI/self-operating-computer
The Self-Operating Computer Framework enables multimodal AI models to autonomously operate a computer by interpreting screen visuals and executing mouse and keyboard actions to achieve user-defined objectives.
wonderwhy-er/DesktopCommanderMCP
Desktop Commander MCP is a server that integrates with the Claude desktop app to provide terminal control, file system search, and diff-based file editing capabilities, enabling advanced AI-assisted desktop management.
jacklandrin/OnlySwitch
OnlySwitch is a customizable all-in-one macOS menu bar app that consolidates system toggles, shortcuts, and AI-powered controls to enhance user productivity and convenience.
iflytek/astron-rpa
AstronRPA is an enterprise-grade open-source Robotic Process Automation desktop application that enables low-code/no-code workflow automation for desktop and web applications with strong enterprise security, collaboration, and AI agent integration.
yuruotong1/autoMate
autoMate is an AI-driven local automation assistant that uses natural language commands to enable users to automate complex computer tasks without programming, supporting local deployment and full interface control for enhanced privacy and flexibility.