OpenNSWM-Lab/FAROS
Blueprint-driven AutoResearch runtime that orchestrates an LLM research workflow from idea refinement through experiment scaffolding, LaTeX paper drafting and simulated peer review.
Awesome AI Agents › Research Platforms & Simulators
facebookresearch/meta-agents-research-environments
Meta Agents Research Environments, abbreviated ARE, is a platform from Meta's FAIR organisation for building and running simulated environments in which AI agents are evaluated on complex, realistic tasks. Its argument is that static benchmarks with a fixed question and a fixed answer no longer measure what matters, because real work changes while it is being done. ARE environments evolve over time: new information arrives, conditions change, and an agent must notice and adapt its plan rather than execute a route decided at the start. Tasks are deliberately long, often requiring ten or more steps and several minutes of wall-clock time, and they are grounded in situations that resemble everyday use of applications and services. The core abstractions documented for the platform are agents, environments, apps, events and scenarios, which together let a researcher script what exists in a world, what happens in it and when. The platform hosts Gaia2, a successor to the Gaia benchmark, comprising 800 scenarios spread across ten simulated universes and covering a broader range of agent capabilities than its predecessor; a separate container-based Gaia2 command-line stack is included for reproducible benchmark runs. Results can be self-published to a public leaderboard on Hugging Face, and an interactive demo runs in the browser without installation. The package is on PyPI and can be run directly with uvx or installed with pip, with a minimal dependency set for command-line benchmarking and an optional GUI extra that adds a web interface for exploring scenarios interactively. Commands cover running a single custom scenario with a chosen agent and running the Gaia2 suite from a Hugging Face dataset split. The intended users are researchers evaluating agent architectures and teams that need a harness for measuring multi-step reasoning and adaptation.
https://github.com/facebookresearch/meta-agents-research-environments
Also in Research Platforms & Simulators
WooooDyy/AgentGym-RL
Framework for training LLM agents on long-horizon, multi-turn decision making with reinforcement learning across a broad set of interactive environments, plus the ScalingInter-RL training method.
Purewhiter/mobilegym
MobileGym is a browser-hosted, fully programmable Android-style simulation platform with 28 simulated apps and 416 task templates for benchmarking and reinforcement-learning training of mobile GUI agents.
SimWorld-AI/SimWorld
SimWorld is an Unreal Engine 5 based open-ended simulator for developing and evaluating LLM and VLM agents in realistic physical and social environments such as procedurally generated cities.
Tencent/AICGSecEval
A.S.E is Tencent's repository-level benchmark that measures the security of AI-generated code, building tasks from real GitHub projects and CVE patches and scoring output with static and dynamic analysis.
HUST-AI-HYZ/MemoryAgentBench
Benchmark and evaluation harness that measures memory in LLM agents across accurate retrieval, test-time learning, long-range understanding and conflict resolution using incremental multi-turn interactions.
DigiRL-agent/digirl
DigiRL is a research project providing code and resources for training autonomous reinforcement learning agents to control Android devices in real-world environments using novel training algorithms and multiple training modes.
xlang-ai/Spider2-V
Spider2-V is a NeurIPS 2024 research project providing a virtual machine environment and dataset to evaluate multimodal agents' ability to automate data science and engineering workflows using state-of-the-art vision-language models.