google-deepmind/acme
Acme is a flexible and scalable research framework providing modular reinforcement learning components and agents for developing and benchmarking RL algorithms.
Awesome AI Agents › Multimodal Model Benchmarks
Ayanami0730/deep_research_bench
DeepResearch Bench is a benchmark for deep research agents, systems that answer a research question by searching, reading sources and writing a long report. The repository contains the task set, the evaluation code and the tooling needed to submit results to a public leaderboard hosted on Hugging Face, and the accompanying paper is on arXiv. Evaluation has two halves. RACE judges report quality with an LLM evaluator against reference material, producing an overall score plus component scores that the maintainers report separately, including comprehensiveness, insight, instruction following and readability aggregated into published metrics such as PAR, OPC, FAP and FAS. FACT evaluates the factual grounding of the report by extracting statement and citation pairs and checking whether the cited source actually supports the claim, which yields citation accuracy and effective citation counts. Because both halves rely on an LLM judge, the maintainers validate evaluator choice against a human-annotated subset and report how closely each candidate judge agrees with human annotators relative to the human inter-annotator agreement baseline; the official evaluator has been migrated between frontier models as older ones were deprecated, with the legacy pipeline preserved on a separate branch and a dual-acceptance window during migration so rankings stay comparable. The evaluation pipeline was revised to clean very long articles in chunks. Running an evaluation requires configuring API keys for the judge and search components as described in the repository configuration section, then executing the pipeline over the generated articles for a target agent. The project comes from Agent Research Lab, which also publishes a successor benchmark, DeepResearch Bench II, with a different evaluation focus based on fine-grained binary rubrics, while stating that the original benchmark continues to be maintained.
https://github.com/Ayanami0730/deep_research_bench
Also in Multimodal Model Benchmarks
THUDM/AgentBench
AgentBench is a comprehensive benchmark platform designed to evaluate large language models as autonomous agents across diverse environments and tasks, facilitating research and development in LLM-based agent capabilities.
microsoft/WindowsAgentArena
Windows Agent Arena is a scalable platform for testing and benchmarking multi-modal AI agents in a realistic Windows OS environment using Azure ML for large-scale deployment.
WooooDyy/AgentGym
AgentGym is a versatile framework for developing, evaluating, and evolving large language model-based agents across diverse interactive environments with real-time feedback and scalability.
TheAgentCompany/TheAgentCompany
TheAgentCompany is a benchmarking platform that evaluates large language model agents on diverse real-world professional tasks within a simulated software company environment to assess their performance and impact on work-related activities.
eleurent/rl-agents
rl-agents is a comprehensive collection of reinforcement learning and planning algorithms with tools for experiment management, benchmarking, and performance monitoring in gym-compatible environments.
LehengTHU/Agent4Rec
Agent4Rec is a recommender system simulator using 1,000 LLM-powered generative agents initialized from MovieLens-1M to simulate realistic user interactions with personalized movie recommendations.
web-arena-x/visualwebarena
VisualWebArena is a benchmark for evaluating autonomous multimodal language agents on complex and realistic web-based visual tasks.