google-deepmind/acme
Acme is a flexible and scalable research framework providing modular reinforcement learning components and agents for developing and benchmarking RL algorithms.
Awesome AI Agents › Multimodal Model Benchmarks
THUDM/VisualAgentBench
VisualAgentBench (VAB) is a pioneering benchmark designed to systematically evaluate and develop large multimodal models (LMMs) as visual foundation agents. It encompasses five distinct environments across three representative visual agent task types: Embodied, GUI, and Visual Design. These environments include VAB-OmniGibson and VAB-Minecraft for embodied tasks, VAB-Mobile and VAB-WebArena-Lite for GUI tasks, and VAB-CSS for visual design tasks. VAB builds upon its predecessor, AgentBench, by emphasizing visual inputs and enabling the development of foundation agent capabilities through training open large language and multimodal models on task trajectories. The framework leverages AgentBench's backbone for efficient and reliable parallel agent evaluation, utilizing an Agent-Controller to interface with LLMs/LMMs and a Task-Controller to access environments. An Assigner component optimizes evaluation speed by pairing agents and tasks. VAB provides a comprehensive setup guide, including environment-specific prerequisites, agent configuration, and task launching procedures. It supports multitask finetuning on its training datasets, which include trajectory data for behavior cloning, allowing the development of more capable visual foundation agents. The benchmark also features a leaderboard showcasing task success rates, comparing proprietary LMMs using prompting and open LMMs after finetuning. VAB integrates and builds upon several notable repositories such as AgentBench, WebArena, OmniGibson, JARVIS-1, and STEVE-1, reflecting a collaborative effort to advance visual agent research. This project is valuable for researchers and developers aiming to push the boundaries of large multimodal models in visual agent tasks, providing a structured and scalable evaluation platform with diverse task environments and training resources.
https://github.com/THUDM/VisualAgentBench
Also in Multimodal Model Benchmarks
THUDM/AgentBench
AgentBench is a comprehensive benchmark platform designed to evaluate large language models as autonomous agents across diverse environments and tasks, facilitating research and development in LLM-based agent capabilities.
microsoft/WindowsAgentArena
Windows Agent Arena is a scalable platform for testing and benchmarking multi-modal AI agents in a realistic Windows OS environment using Azure ML for large-scale deployment.
WooooDyy/AgentGym
AgentGym is a versatile framework for developing, evaluating, and evolving large language model-based agents across diverse interactive environments with real-time feedback and scalability.
Ayanami0730/deep_research_bench
Benchmark and evaluation pipeline for deep research agents, scoring long-form generated reports on quality with the RACE framework and on citation trustworthiness with the FACT framework, plus a public leaderboard.
TheAgentCompany/TheAgentCompany
TheAgentCompany is a benchmarking platform that evaluates large language model agents on diverse real-world professional tasks within a simulated software company environment to assess their performance and impact on work-related activities.
eleurent/rl-agents
rl-agents is a comprehensive collection of reinforcement learning and planning algorithms with tools for experiment management, benchmarking, and performance monitoring in gym-compatible environments.
LehengTHU/Agent4Rec
Agent4Rec is a recommender system simulator using 1,000 LLM-powered generative agents initialized from MovieLens-1M to simulate realistic user interactions with personalized movie recommendations.