OpenNSWM-Lab/FAROS
Blueprint-driven AutoResearch runtime that orchestrates an LLM research workflow from idea refinement through experiment scaffolding, LaTeX paper drafting and simulated peer review.
Awesome AI Agents › Research Platforms & Simulators
Tencent/AICGSecEval
A.S.E, short for AICGSecEval, is a benchmark and evaluation framework for the security of AI-generated code, published by the Tencent Wukong Code Security Team. Unlike function-level coding benchmarks, it works at the repository level and tries to reproduce how AI-assisted programming actually happens in a real project. The task set is derived from real GitHub repositories and from authoritative CVE patches, so each generation task sits at a place in a codebase where a genuine vulnerability once existed and where a security-relevant decision has to be made. During generation the framework automatically extracts project-level code context and supplies it to the model or agent under test, simulating a realistic development scenario rather than an isolated prompt. Evaluation then combines static and dynamic analysis in a hybrid suite, trading off broad detection coverage against precise verification so that reported results are reproducible and not driven by a single scanner's false positives. The 2.0 release broadens the dataset to cover more code generation vulnerability scenarios, including risks drawn from the OWASP Top 10 and the CWE Top 25. The project is written in Python, ships a quick-start path for running evaluations locally, and publishes results through a public leaderboard site along with news updates and an accompanying arXiv paper that documents the methodology. The maintainers position it as an open, reproducible and continuously evolving community project and invite contributions that expand the dataset or improve the evaluation harness. It is intended for researchers studying the security properties of code-generating models and agents, for teams choosing between coding assistants, and for model developers who need a security-focused regression measure alongside functional coding benchmarks.
https://github.com/Tencent/AICGSecEval
Also in Research Platforms & Simulators
WooooDyy/AgentGym-RL
Framework for training LLM agents on long-horizon, multi-turn decision making with reinforcement learning across a broad set of interactive environments, plus the ScalingInter-RL training method.
Purewhiter/mobilegym
MobileGym is a browser-hosted, fully programmable Android-style simulation platform with 28 simulated apps and 416 task templates for benchmarking and reinforcement-learning training of mobile GUI agents.
SimWorld-AI/SimWorld
SimWorld is an Unreal Engine 5 based open-ended simulator for developing and evaluating LLM and VLM agents in realistic physical and social environments such as procedurally generated cities.
facebookresearch/meta-agents-research-environments
Meta's research platform for evaluating AI agents in dynamic simulated environments, hosting the Gaia2 benchmark of 800 evolving scenarios across ten universes, with a CLI, a web GUI and a public leaderboard.
HUST-AI-HYZ/MemoryAgentBench
Benchmark and evaluation harness that measures memory in LLM agents across accurate retrieval, test-time learning, long-range understanding and conflict resolution using incremental multi-turn interactions.
DigiRL-agent/digirl
DigiRL is a research project providing code and resources for training autonomous reinforcement learning agents to control Android devices in real-world environments using novel training algorithms and multiple training modes.
xlang-ai/Spider2-V
Spider2-V is a NeurIPS 2024 research project providing a virtual machine environment and dataset to evaluate multimodal agents' ability to automate data science and engineering workflows using state-of-the-art vision-language models.