bytedance/UI-TARS-desktop
UI-TARS Desktop is a GUI Agent application that uses a Vision-Language Model to enable natural language control of computers across multiple platforms with real-time visual recognition and automation capabilities.
Awesome AI Agents › Desktop Automation
niuzaisheng/ScreenAgent
ScreenAgent is a project that creates a computer control agent driven by a Visual Language Large Model (VLM). The agent interacts with real computer screens by observing screenshots and manipulating the GUI through mouse and keyboard operations. The project introduces an automatic control process consisting of planning, action, and reflection stages, enabling the agent to continuously interact with the environment and complete multi-step tasks. The ScreenAgent dataset, manually annotated, includes screenshots and action sequences for various daily computer tasks such as file operations, web browsing, and gaming. The agent's action space is designed based on the VNC remote desktop protocol, supporting basic mouse and keyboard operations with precise screen coordinate inputs, making it applicable across different desktop operating systems and applications. The project structure includes client code for controlling the environment, datasets for training and vision positioning, model workers for VLM inference, and training code. Users need to prepare a desktop environment with a VNC server or use a provided Docker container to run the controlled desktop. The controller code connects to the VNC server, collects screenshots, sends control commands, and manages the agent's interaction process. It also constructs prompts for the large model inference API and parses the generated control commands. The project supports multiple VLMs like GPT-4V, LLaVA-1.5, CogAgent, and ScreenAgent itself, with options to implement custom inferencers or use third-party APIs. The system includes a clipboard service to facilitate text input, especially for unicode characters, enhancing the agent's ability to perform complex tasks. Overall, ScreenAgent aims to teach a VLM agent to use a computer by integrating task planning, image understanding, visual positioning, and tool use, providing a universal and extensible framework for automated computer control.
https://github.com/niuzaisheng/ScreenAgent
Also in Desktop Automation
asweigart/pyautogui
PyAutoGUI is a cross-platform Python module that enables programmatic control of the mouse and keyboard for GUI automation, including features like screen capture and image recognition.
bytebot-ai/bytebot
Bytebot is a containerized Linux desktop environment that enables programmatic control and remote access for automation, scraping, CI tasks, and remote work through a unified REST API and VNC interfaces.
OthersideAI/self-operating-computer
The Self-Operating Computer Framework enables multimodal AI models to autonomously operate a computer by interpreting screen visuals and executing mouse and keyboard actions to achieve user-defined objectives.
wonderwhy-er/DesktopCommanderMCP
Desktop Commander MCP is a server that integrates with the Claude desktop app to provide terminal control, file system search, and diff-based file editing capabilities, enabling advanced AI-assisted desktop management.
jacklandrin/OnlySwitch
OnlySwitch is a customizable all-in-one macOS menu bar app that consolidates system toggles, shortcuts, and AI-powered controls to enhance user productivity and convenience.
iflytek/astron-rpa
AstronRPA is an enterprise-grade open-source Robotic Process Automation desktop application that enables low-code/no-code workflow automation for desktop and web applications with strong enterprise security, collaboration, and AI agent integration.
yuruotong1/autoMate
autoMate is an AI-driven local automation assistant that uses natural language commands to enable users to automate complex computer tasks without programming, supporting local deployment and full interface control for enhanced privacy and flexibility.