always-further/deepfabric
DeepFabric generates synthetic training and evaluation datasets for agentic models, combining topic graph expansion, reasoning traces, constrained decoding and real tool execution in isolated WebAssembly sandboxes.
Awesome AI Agents › Training Datasets
Dongping-Chen/GUI-World
GUI-World is a comprehensive dataset and benchmark designed for evaluating multimodal large language models (MLLMs) in the context of graphical user interfaces (GUIs). It addresses the challenge of understanding and interacting with dynamic and complex GUI environments, which are common in real-world applications. The dataset includes extensive annotations across six different GUI scenarios and eight types of GUI-oriented questions, providing a rich resource for training and testing models on tasks that require perception and interaction with both static and dynamic GUI elements. GUI-World aims to highlight the limitations of current state-of-the-art ImageLLMs and VideoLLMs in handling multi-step and dynamic tasks within GUIs, offering valuable insights for future research and development in this area. The project also introduces GUI-Vid, a GUI-oriented Video Large Language Model (VideoLLM) fine-tuned from the Videochat2 model. GUI-Vid is designed to perform inference on GUI videos, enabling the analysis and understanding of GUI interactions over time. The repository provides detailed instructions for setting up the environment, fine-tuning the model with the GUI-World dataset, and running inference on GUI videos. Key features include multiple methods for extracting keyframes from videos, such as uniform sampling, programmatic extraction using Katna, and additional experiments using VIP and R3M frameworks. The project is actively maintained, with recent updates including the release of benchmark code and a related research paper available on arXiv. Contributions to the project are encouraged, particularly in proposing new features, improvements, or benchmarking other mainstream MLLMs. Overall, GUI-World serves as a foundational resource for advancing the capabilities of multimodal LLMs in understanding and interacting with GUIs, facilitating the development of more robust and intelligent GUI agents.
https://github.com/Dongping-Chen/GUI-World
Also in Training Datasets
OpenGVLab/GUI-Odyssey
GUI Odyssey is a comprehensive dataset for training and evaluating cross-app navigation agents on mobile devices, featuring 7,735 episodes across multiple devices, apps, and navigation tasks.
neulab/MultiUI
MultiUI is a large-scale dataset and codebase for training and evaluating models on text-rich visual understanding of webpage user interfaces, featuring 7.3 million samples, pre-trained models, and comprehensive benchmark evaluations.