google-deepmind/acme
Acme is a flexible and scalable research framework providing modular reinforcement learning components and agents for developing and benchmarking RL algorithms.
Awesome AI Agents › Multimodal Model Benchmarks
THUDM/AgentBench
AgentBench is a comprehensive benchmark designed to evaluate large language models (LLMs) as autonomous agents across a diverse range of environments. It provides a systematic framework to assess the capabilities of LLMs in operating as agents in various complex scenarios. The benchmark includes eight distinct environments, five of which are newly created domains such as Operating System (OS), Database (DB), Knowledge Graph (KG), Digital Card Game (DCG), and Lateral Thinking Puzzles (LTP). Additionally, it incorporates three environments recompiled from existing datasets: House-Holding (HH) from ALFWorld, Web Shopping (WS) from WebShop, and Web Browsing (WB) from Mind2Web. AgentBench offers two dataset splits, Dev and Test, requiring multi-turn interactions where LLMs generate thousands of responses to complete tasks. The benchmark also features a leaderboard to track model performance, highlighting the current gaps between models and practical usability. The project provides detailed setup instructions, including environment configuration, Docker image building, and task launching, facilitating easy adoption and experimentation. A notable extension is VisualAgentBench, which evaluates and trains visual foundation agents based on large multimodal models (LMMs) across five environments spanning embodied, GUI, and visual design tasks. VisualAgentBench benchmarks 17 proprietary and open LMMs and offers trajectory datasets for behavior cloning training. AgentBench supports extensibility for adding new tasks and provides resources for deploying dependent services like KnowledgeGraph locally. The project is actively maintained with community engagement through Slack, a Google Group, and social media, and it is backed by a research paper published at ICLR 2024. Overall, AgentBench serves as a vital tool for advancing research and development in autonomous LLM agents by providing a robust, diverse, and extensible evaluation platform.
https://github.com/THUDM/AgentBench
Also in Multimodal Model Benchmarks
microsoft/WindowsAgentArena
Windows Agent Arena is a scalable platform for testing and benchmarking multi-modal AI agents in a realistic Windows OS environment using Azure ML for large-scale deployment.
WooooDyy/AgentGym
AgentGym is a versatile framework for developing, evaluating, and evolving large language model-based agents across diverse interactive environments with real-time feedback and scalability.
Ayanami0730/deep_research_bench
Benchmark and evaluation pipeline for deep research agents, scoring long-form generated reports on quality with the RACE framework and on citation trustworthiness with the FACT framework, plus a public leaderboard.
TheAgentCompany/TheAgentCompany
TheAgentCompany is a benchmarking platform that evaluates large language model agents on diverse real-world professional tasks within a simulated software company environment to assess their performance and impact on work-related activities.
eleurent/rl-agents
rl-agents is a comprehensive collection of reinforcement learning and planning algorithms with tools for experiment management, benchmarking, and performance monitoring in gym-compatible environments.
LehengTHU/Agent4Rec
Agent4Rec is a recommender system simulator using 1,000 LLM-powered generative agents initialized from MovieLens-1M to simulate realistic user interactions with personalized movie recommendations.
web-arena-x/visualwebarena
VisualWebArena is a benchmark for evaluating autonomous multimodal language agents on complex and realistic web-based visual tasks.