google-deepmind/acme
Acme is a flexible and scalable research framework providing modular reinforcement learning components and agents for developing and benchmarking RL algorithms.
Awesome AI Agents › Multimodal Model Benchmarks
microsoft/WindowsAgentArena
Windows Agent Arena (WAA) is a scalable and comprehensive platform designed for testing and benchmarking multi-modal AI agents within a realistic Windows operating system environment. It provides researchers and developers with a reproducible setup to evaluate agentic AI workflows across a wide variety of tasks on the Windows desktop. The platform supports large-scale deployment of AI agents using Azure ML cloud infrastructure, enabling parallel execution of multiple agents and rapid benchmarking results for hundreds of tasks within minutes rather than days. WAA includes a Windows 11 virtual machine snapshot, known as the golden image, which is pre-configured with necessary programs and a Python server to execute agent commands. This setup allows agents to interact with the Windows OS environment in a controlled and realistic manner. The platform supports different difficulty modes, including a harder mode where agents must autonomously initialize and set up tasks, such as finding and opening the correct applications, enhancing the challenge and realism of the testing environment. WAA also integrates with OpenAI and Azure OpenAI APIs to leverage advanced AI models. The project provides detailed instructions for local deployment using Docker and WSL 2, including building the necessary Docker images and preparing the Windows 11 VM. The platform is open-source under the MIT license and is accompanied by a technical report published on arXiv, a project website, and additional resources such as a podcast and blog post. WAA aims to advance research in multi-modal AI agents by providing a scalable, realistic, and reproducible environment for benchmarking AI performance on complex desktop tasks.
https://github.com/microsoft/WindowsAgentArena
Also in Multimodal Model Benchmarks
THUDM/AgentBench
AgentBench is a comprehensive benchmark platform designed to evaluate large language models as autonomous agents across diverse environments and tasks, facilitating research and development in LLM-based agent capabilities.
WooooDyy/AgentGym
AgentGym is a versatile framework for developing, evaluating, and evolving large language model-based agents across diverse interactive environments with real-time feedback and scalability.
Ayanami0730/deep_research_bench
Benchmark and evaluation pipeline for deep research agents, scoring long-form generated reports on quality with the RACE framework and on citation trustworthiness with the FACT framework, plus a public leaderboard.
TheAgentCompany/TheAgentCompany
TheAgentCompany is a benchmarking platform that evaluates large language model agents on diverse real-world professional tasks within a simulated software company environment to assess their performance and impact on work-related activities.
eleurent/rl-agents
rl-agents is a comprehensive collection of reinforcement learning and planning algorithms with tools for experiment management, benchmarking, and performance monitoring in gym-compatible environments.
LehengTHU/Agent4Rec
Agent4Rec is a recommender system simulator using 1,000 LLM-powered generative agents initialized from MovieLens-1M to simulate realistic user interactions with personalized movie recommendations.
web-arena-x/visualwebarena
VisualWebArena is a benchmark for evaluating autonomous multimodal language agents on complex and realistic web-based visual tasks.