google-deepmind/acme
Acme is a flexible and scalable research framework providing modular reinforcement learning components and agents for developing and benchmarking RL algorithms.
Awesome AI Agents › Multimodal Model Benchmarks
camel-ai/crab
CRAB (Cross-environment Agent Benchmark) is a comprehensive framework designed to build and benchmark multimodal embodied language model agents across diverse environments. It provides a Python-centric approach to creating agent environments that are cross-platform and support multiple deployment options such as in-memory, Docker-hosted, virtual machines, or distributed physical machines, as long as they are accessible via Python functions. This flexibility allows agents to interact with various environments simultaneously through a unified interface, facilitating extensive testing and evaluation. The framework emphasizes ease of use with a simple configuration system where new actions can be added by decorating Python functions with an @action decorator. Environments are defined by integrating these actions, making it straightforward to customize and extend the framework for different use cases. CRAB also introduces a novel benchmarking suite that allows users to define tasks and evaluators in an intuitive Python-native manner. A unique graph evaluator method is included to provide fine-grained metrics, enhancing the depth and quality of agent performance evaluation. CRAB supports experimentation with a dedicated benchmark dataset and experiment code available in the crab-benchmark-v0 directory. It is compatible with Python 3.10 or newer and can be installed easily via pip. The project includes example scripts demonstrating how to run template environments with OpenAI agents, showcasing practical applications of the framework. The project is well-documented with accessible resources including a website, demos, blog posts, and a demo video. It is actively maintained and supported by a community with channels on Slack, Discord, WeChat, and Twitter. CRAB is also academically recognized, with a related research paper published on arXiv, providing a solid foundation for its methodologies and innovations. Overall, CRAB is a powerful tool for researchers and developers working on multimodal language model agents, offering a versatile, easy-to-use, and robust benchmarking environment that supports cross-platform deployment and detailed performance analysis.
https://github.com/camel-ai/crab
Also in Multimodal Model Benchmarks
THUDM/AgentBench
AgentBench is a comprehensive benchmark platform designed to evaluate large language models as autonomous agents across diverse environments and tasks, facilitating research and development in LLM-based agent capabilities.
microsoft/WindowsAgentArena
Windows Agent Arena is a scalable platform for testing and benchmarking multi-modal AI agents in a realistic Windows OS environment using Azure ML for large-scale deployment.
WooooDyy/AgentGym
AgentGym is a versatile framework for developing, evaluating, and evolving large language model-based agents across diverse interactive environments with real-time feedback and scalability.
Ayanami0730/deep_research_bench
Benchmark and evaluation pipeline for deep research agents, scoring long-form generated reports on quality with the RACE framework and on citation trustworthiness with the FACT framework, plus a public leaderboard.
TheAgentCompany/TheAgentCompany
TheAgentCompany is a benchmarking platform that evaluates large language model agents on diverse real-world professional tasks within a simulated software company environment to assess their performance and impact on work-related activities.
eleurent/rl-agents
rl-agents is a comprehensive collection of reinforcement learning and planning algorithms with tools for experiment management, benchmarking, and performance monitoring in gym-compatible environments.
LehengTHU/Agent4Rec
Agent4Rec is a recommender system simulator using 1,000 LLM-powered generative agents initialized from MovieLens-1M to simulate realistic user interactions with personalized movie recommendations.