google-deepmind/acme
Acme is a flexible and scalable research framework providing modular reinforcement learning components and agents for developing and benchmarking RL algorithms.
Awesome AI Agents › Multimodal Model Benchmarks
shulin16/MMInA
MMInA (Benchmarking Multihop Multimodal Internet Agents) is a research project that provides an official implementation of a benchmark designed to evaluate the long-chain reasoning capabilities of multimodal internet agents. The project is centered around a dataset and environment that challenge agents to perform multihop reasoning tasks involving multimodal information, such as text and images, across various domains. The dataset is structured into multiple subsets categorized by the number of reasoning hops required, ranging from single-hop tasks to complex tasks requiring up to ten hops. These subsets include domains like shopping, Wikipedia, and other specialized categories that test different aspects of multimodal understanding and reasoning. The project builds upon the WebArena environment, a self-hostable web environment for autonomous agents that interact through textual inputs. MMInA extends this environment to support multimodal tasks and provides a comprehensive codebase, dataset, and evaluation framework. Users can install the environment using provided scripts and dependencies, and they can run pre-developed agents or implement their own customized agents to test on the benchmark tasks. MMInA supports agents with and without history memory, allowing for experiments on how memory affects reasoning performance. The project includes detailed instructions for environment setup, dataset download, and running experiments. It also offers flexibility for researchers to customize agents by modifying the agent implementation and configuration files. The benchmark aims to push the boundaries of multimodal AI by providing a challenging and diverse set of tasks that require integrating information from multiple modalities and reasoning steps. The project is accompanied by a research paper, dataset, and codebase released publicly to facilitate further research in this area. Overall, MMInA serves as a valuable resource for advancing the development and evaluation of intelligent multimodal agents capable of complex internet-based reasoning.
https://github.com/shulin16/MMInA
Also in Multimodal Model Benchmarks
THUDM/AgentBench
AgentBench is a comprehensive benchmark platform designed to evaluate large language models as autonomous agents across diverse environments and tasks, facilitating research and development in LLM-based agent capabilities.
microsoft/WindowsAgentArena
Windows Agent Arena is a scalable platform for testing and benchmarking multi-modal AI agents in a realistic Windows OS environment using Azure ML for large-scale deployment.
WooooDyy/AgentGym
AgentGym is a versatile framework for developing, evaluating, and evolving large language model-based agents across diverse interactive environments with real-time feedback and scalability.
Ayanami0730/deep_research_bench
Benchmark and evaluation pipeline for deep research agents, scoring long-form generated reports on quality with the RACE framework and on citation trustworthiness with the FACT framework, plus a public leaderboard.
TheAgentCompany/TheAgentCompany
TheAgentCompany is a benchmarking platform that evaluates large language model agents on diverse real-world professional tasks within a simulated software company environment to assess their performance and impact on work-related activities.
eleurent/rl-agents
rl-agents is a comprehensive collection of reinforcement learning and planning algorithms with tools for experiment management, benchmarking, and performance monitoring in gym-compatible environments.
LehengTHU/Agent4Rec
Agent4Rec is a recommender system simulator using 1,000 LLM-powered generative agents initialized from MovieLens-1M to simulate realistic user interactions with personalized movie recommendations.