google-deepmind/acme
Acme is a flexible and scalable research framework providing modular reinforcement learning components and agents for developing and benchmarking RL algorithms.
Awesome AI Agents › Multimodal Model Benchmarks
ltzheng/agent-studio
AgentStudio is a comprehensive toolkit designed to facilitate the development, evaluation, and benchmarking of general virtual agents capable of interacting with any computer software. It provides a trinity of environments, tools, and benchmarks aimed at creating robust, general, and open-ended virtual agents. The environment is lightweight and interactive, featuring highly generic observation and action spaces such as video observations and GUI/API actions, which significantly expand the task space for agents. AgentStudio includes tools for creating online benchmark tasks, annotating GUI elements, and labeling actions in videos, enabling detailed and structured data generation for training and evaluation purposes. The toolkit offers a suite of 205 online benchmark tasks that cover a wide range of API usages and GUI software, including terminal commands, Gmail, and VS Code, with auto-evaluation and language feedback to assess agent performance. Additionally, AgentStudio provides three benchmark datasets—GroundUI, IDMBench, and CriticBench—that focus on fundamental agent abilities like GUI grounding, learning from videos, and success detection. These datasets help in gaining deeper insights into agent capabilities beyond overall task performance. The project also supplies tools for benchmark task creation and validation, step-level GUI element annotation, and trajectory-level video-action recording and refinement. AgentStudio is actively maintained with regular updates and comprehensive documentation, and it encourages community contributions. It is licensed under AGPL v3 and supports integration with various APIs through configurable keys. The project is well-suited for researchers and developers working on virtual agents, human-computer interaction, and AI benchmarking, providing a rich platform for advancing the state of general virtual agents.
https://github.com/ltzheng/agent-studio
Also in Multimodal Model Benchmarks
THUDM/AgentBench
AgentBench is a comprehensive benchmark platform designed to evaluate large language models as autonomous agents across diverse environments and tasks, facilitating research and development in LLM-based agent capabilities.
microsoft/WindowsAgentArena
Windows Agent Arena is a scalable platform for testing and benchmarking multi-modal AI agents in a realistic Windows OS environment using Azure ML for large-scale deployment.
WooooDyy/AgentGym
AgentGym is a versatile framework for developing, evaluating, and evolving large language model-based agents across diverse interactive environments with real-time feedback and scalability.
Ayanami0730/deep_research_bench
Benchmark and evaluation pipeline for deep research agents, scoring long-form generated reports on quality with the RACE framework and on citation trustworthiness with the FACT framework, plus a public leaderboard.
TheAgentCompany/TheAgentCompany
TheAgentCompany is a benchmarking platform that evaluates large language model agents on diverse real-world professional tasks within a simulated software company environment to assess their performance and impact on work-related activities.
eleurent/rl-agents
rl-agents is a comprehensive collection of reinforcement learning and planning algorithms with tools for experiment management, benchmarking, and performance monitoring in gym-compatible environments.
LehengTHU/Agent4Rec
Agent4Rec is a recommender system simulator using 1,000 LLM-powered generative agents initialized from MovieLens-1M to simulate realistic user interactions with personalized movie recommendations.