Awesome AI Agents › Browser Automation Tools

OSU-NLP-Group/Mind2Web

⭐ 1037 Jupyter Notebook added to this list on 2025-04-19 repository created 2023-04-21

Mind2Web is a pioneering dataset and framework designed to develop and evaluate generalist web agents capable of following natural language instructions to perform complex tasks across a wide variety of real-world websites. Unlike previous datasets that rely on simulated or limited website environments, Mind2Web encompasses over 2,000 open-ended tasks collected from 137 websites spanning 31 diverse domains, providing a rich and realistic environment for training and testing web agents. The dataset includes crowdsourced action sequences that detail the steps required to complete each task, capturing a broad spectrum of user interaction patterns such as clicking, typing, and selecting elements on web pages. The project offers a comprehensive dataset structure with training and multiple test splits designed to evaluate generalization across tasks, websites, and domains. The training set contains over a thousand instances, while the test sets are divided into cross-task, cross-website, and cross-domain splits to rigorously assess the agent's ability to handle unseen tasks and environments. Each data instance includes detailed annotations such as unique task IDs, website and domain information, task descriptions, and step-by-step action sequences with corresponding HTML snapshots before each action. Mind2Web also provides raw data dumps with full traces, network traffic captures, video recordings, and snapshots to facilitate in-depth analysis and replay of user interactions. The project is actively maintained with updates including fine-tuning code, models, and related datasets like Multimodal-Mind2Web, which pairs HTML documents with webpage screenshots for multimodal learning. Additionally, the project links to follow-up works such as SeeAct, which enables GPT-4V-based web agents. Overall, Mind2Web represents a significant advancement in building versatile and robust web agents that can operate across diverse real-world websites by leveraging a large-scale, richly annotated dataset and comprehensive tooling for training, evaluation, and analysis.

https://github.com/OSU-NLP-Group/Mind2Web

cross-domain-generalizationcross-task-generalizationcross-website-generalizationcrowdsourced-actionsdatasetdiverse-domainsfine-tuninggeneralist-web-agentgpt-4vhtml-snapshotsmind2webmultimodal-datasetnatural-language-instructionsnetwork-trafficraw-data-dumpreal-world-websitesseeactuser-interaction-patternsvideo-recordingsweb-agent-evaluationweb-automationweb-tasks