OpenNSWM-Lab/FAROS
Blueprint-driven AutoResearch runtime that orchestrates an LLM research workflow from idea refinement through experiment scaffolding, LaTeX paper drafting and simulated peer review.
Awesome AI Agents › Research Platforms & Simulators
xlang-ai/Spider2-V
Spider2-V is a cutting-edge research project presented at NeurIPS 2024 that explores the capabilities and limitations of multimodal agents in automating data science and engineering workflows. The project provides a comprehensive environment and dataset designed to evaluate how well state-of-the-art vision-language models (VLMs) can perform complex tasks typically handled by human data scientists and engineers. The environment simulates a desktop Ubuntu 22.04 LTS system within a virtual machine, allowing agents to interact with real software and tools through a graphical user interface. This setup enables realistic testing of agent performance on a variety of data science and engineering tasks. The project includes detailed installation instructions for setting up the environment on desktop or server systems, including dependencies, virtual machine software (VMware Workstation Pro or Fusion), and browser automation tools like Playwright. It supports both ARM64 and x86 architectures with pre-configured virtual machine images available for download. Users can run example scripts to quickly start interacting with the environment and evaluate agent performance. Spider2-V features a diverse set of 494 task examples spanning 20 different applications, categorized by instruction verbosity and account requirements. The project provides tools for both automated and human-in-the-loop evaluation, allowing researchers to validate agent task completion through screenshots, accessibility trees, and video recordings. Baseline agent implementations using advanced models such as GPT-4o and GPT-4V are included, showcasing current success rates and highlighting areas for improvement. Overall, Spider2-V serves as a valuable benchmark and research platform for advancing the development of intelligent multimodal agents capable of automating complex workflows in data science and engineering domains. It combines realistic virtual environments, comprehensive datasets, and state-of-the-art agent baselines to push the boundaries of AI-assisted automation.
https://github.com/xlang-ai/Spider2-V
Also in Research Platforms & Simulators
WooooDyy/AgentGym-RL
Framework for training LLM agents on long-horizon, multi-turn decision making with reinforcement learning across a broad set of interactive environments, plus the ScalingInter-RL training method.
Purewhiter/mobilegym
MobileGym is a browser-hosted, fully programmable Android-style simulation platform with 28 simulated apps and 416 task templates for benchmarking and reinforcement-learning training of mobile GUI agents.
SimWorld-AI/SimWorld
SimWorld is an Unreal Engine 5 based open-ended simulator for developing and evaluating LLM and VLM agents in realistic physical and social environments such as procedurally generated cities.
Tencent/AICGSecEval
A.S.E is Tencent's repository-level benchmark that measures the security of AI-generated code, building tasks from real GitHub projects and CVE patches and scoring output with static and dynamic analysis.
facebookresearch/meta-agents-research-environments
Meta's research platform for evaluating AI agents in dynamic simulated environments, hosting the Gaia2 benchmark of 800 evolving scenarios across ten universes, with a CLI, a web GUI and a public leaderboard.
HUST-AI-HYZ/MemoryAgentBench
Benchmark and evaluation harness that measures memory in LLM agents across accurate retrieval, test-time learning, long-range understanding and conflict resolution using incremental multi-turn interactions.
DigiRL-agent/digirl
DigiRL is a research project providing code and resources for training autonomous reinforcement learning agents to control Android devices in real-world environments using novel training algorithms and multiple training modes.