always-further/deepfabric
DeepFabric generates synthetic training and evaluation datasets for agentic models, combining topic graph expansion, reasoning traces, constrained decoding and real tool execution in isolated WebAssembly sandboxes.
Awesome AI Agents › Training Datasets
neulab/MultiUI
MultiUI is a comprehensive dataset and codebase designed for advancing text-rich visual understanding of webpage user interfaces (UIs). The project provides a large-scale dataset consisting of 7.3 million samples that cover a wide variety of UI types and tasks. These samples are structured using enhanced accessibility trees and task taxonomies, which facilitate detailed and nuanced analysis of UI elements and their interactions. The repository is organized into two main parts: training and evaluation. The training section includes code for training LLaVA-OneVision, the foundational model used in the research. The evaluation section contains scripts and tools to assess model performance across multiple benchmarks relevant to GUI understanding, OCR, document and chart question answering, and general grounding tasks. The dataset, MultiUI, is publicly available for download on Hugging Face, enabling researchers and developers to utilize this extensive resource for their own experiments. The project also provides pre-trained model checkpoints, including UIX-Qwen2 and UIX-Qwen2-Mind2Web, which combine large language models with vision towers to enhance visual and textual comprehension of UIs. Evaluation is supported on various benchmarks such as VisualWebBench, WebSRC, ScreenQA-short, WidgetCap, ScreenSpot, RefExp, DocVQA, ChartQA, TextVQA, InfoVQA, VisualMRC, OCRBench, and RefCOCO+. The repository includes detailed instructions for running these evaluations using provided scripts and the lmms-eval framework. Additionally, there is support for Mind2Web evaluation, which involves downloading a processed dataset and running inference and metric calculations. The dataset is sourced primarily from publicly available web content and generated data, with a focus on open-source research use. The creators acknowledge potential biases and inaccuracies and provide a contact mechanism for content concerns. Overall, MultiUI is a valuable resource for advancing research in multimodal understanding of web interfaces, combining large-scale data, model training, and comprehensive evaluation tools.
https://github.com/neulab/MultiUI
Also in Training Datasets
OpenGVLab/GUI-Odyssey
GUI Odyssey is a comprehensive dataset for training and evaluating cross-app navigation agents on mobile devices, featuring 7,735 episodes across multiple devices, apps, and navigation tasks.
Dongping-Chen/GUI-World
GUI-World is a dataset and benchmark for evaluating and advancing multimodal large language models in understanding and interacting with dynamic graphical user interfaces, complemented by GUI-Vid, a fine-tuned VideoLLM for GUI video analysis.