OpenNSWM-Lab/FAROS
Blueprint-driven AutoResearch runtime that orchestrates an LLM research workflow from idea refinement through experiment scaffolding, LaTeX paper drafting and simulated peer review.
Awesome AI Agents › Research Platforms & Simulators
HUST-AI-HYZ/MemoryAgentBench
MemoryAgentBench is the code and data release for an ICLR 2026 paper on evaluating how well LLM agents remember information across long, incremental conversations. Instead of asking a model a single long-context question, the benchmark splits source material into chunks that are fed to an agent turn by turn, simulating the way information arrives in a real multi-turn session, and then queries the agent about what it retained. Four competencies are measured: accurate retrieval, meaning recovering a specific fact from earlier turns; test-time learning, meaning acquiring and applying new information during the session; long-range understanding, meaning reasoning over material spread across the whole history; and conflict resolution, meaning handling later information that contradicts what was stored before. The dataset combines reformulated material from earlier benchmarks with two new sets built by the authors, EventQA and FactConsolidation, and is published on Hugging Face. An inject-once, query-many design attaches multiple questions to each long input so that a single expensive ingestion supports many evaluations, which cuts the cost of a full run. The harness evaluates several classes of memory system under the same protocol, including plain long-context agents, retrieval-augmented agents and dedicated agentic memory frameworks such as HippoRAG, Cognee and Letta, configured through agent and dataset configuration files and launched from shell scripts. Scoring for summarisation and open-ended tasks uses an LLM judge, with API keys for OpenAI, Anthropic and Google supplied through an environment file. A follow-up project, MemoryArena, extends the same line of work to agentic tasks. The audience is researchers and engineers comparing memory architectures for conversational agents, and authors of memory libraries who want a standard, reproducible measurement rather than ad hoc demonstrations.
https://github.com/HUST-AI-HYZ/MemoryAgentBench
Also in Research Platforms & Simulators
WooooDyy/AgentGym-RL
Framework for training LLM agents on long-horizon, multi-turn decision making with reinforcement learning across a broad set of interactive environments, plus the ScalingInter-RL training method.
Purewhiter/mobilegym
MobileGym is a browser-hosted, fully programmable Android-style simulation platform with 28 simulated apps and 416 task templates for benchmarking and reinforcement-learning training of mobile GUI agents.
SimWorld-AI/SimWorld
SimWorld is an Unreal Engine 5 based open-ended simulator for developing and evaluating LLM and VLM agents in realistic physical and social environments such as procedurally generated cities.
Tencent/AICGSecEval
A.S.E is Tencent's repository-level benchmark that measures the security of AI-generated code, building tasks from real GitHub projects and CVE patches and scoring output with static and dynamic analysis.
facebookresearch/meta-agents-research-environments
Meta's research platform for evaluating AI agents in dynamic simulated environments, hosting the Gaia2 benchmark of 800 evolving scenarios across ten universes, with a CLI, a web GUI and a public leaderboard.
DigiRL-agent/digirl
DigiRL is a research project providing code and resources for training autonomous reinforcement learning agents to control Android devices in real-world environments using novel training algorithms and multiple training modes.
xlang-ai/Spider2-V
Spider2-V is a NeurIPS 2024 research project providing a virtual machine environment and dataset to evaluate multimodal agents' ability to automate data science and engineering workflows using state-of-the-art vision-language models.