google-deepmind/acme
Acme is a flexible and scalable research framework providing modular reinforcement learning components and agents for developing and benchmarking RL algorithms.
Awesome AI Agents › Multimodal Model Benchmarks
MobileAgentBench/mobile-agent-bench
MobileAgentBench is an automated benchmarking framework designed specifically for evaluating mobile large language model (LLM) agents. The project facilitates performance testing of mobile agents by running them through a series of predefined tasks using popular mobile applications. It leverages Android emulators or physical devices to simulate real-world usage scenarios, providing a controlled environment for benchmarking. The benchmark tasks utilize apps from SimpleMobileTools, including Calculator, Calendar, Contacts, FileManager, Gallery, AppLauncher, Messager, MusicPlayer, Notes, and Recorder, which are installed on the testing device to serve as the operational context for the agents. The setup process involves installing Android Studio, which provides essential debugging tools such as ADB and Android emulators. Users are guided to configure environment variables to ensure proper toolchain functionality. The benchmark itself is packaged as a Python library, allowing easy integration into other projects. Users can build and install the library from source, enabling them to import and use the benchmarking tools programmatically. For development and testing purposes, the project includes a dummy agent script that simulates agent behavior by simply sleeping in a loop. This helps verify that the benchmarking framework correctly detects task completion without requiring a fully functional agent implementation. Overall, MobileAgentBench provides a structured and automated approach to assess the capabilities and performance of mobile LLM agents across a variety of common mobile app tasks. It supports developers and researchers in benchmarking their agents in a consistent and reproducible manner, facilitating improvements and comparisons in mobile AI agent technology.
https://github.com/MobileAgentBench/mobile-agent-bench
Also in Multimodal Model Benchmarks
THUDM/AgentBench
AgentBench is a comprehensive benchmark platform designed to evaluate large language models as autonomous agents across diverse environments and tasks, facilitating research and development in LLM-based agent capabilities.
microsoft/WindowsAgentArena
Windows Agent Arena is a scalable platform for testing and benchmarking multi-modal AI agents in a realistic Windows OS environment using Azure ML for large-scale deployment.
WooooDyy/AgentGym
AgentGym is a versatile framework for developing, evaluating, and evolving large language model-based agents across diverse interactive environments with real-time feedback and scalability.
Ayanami0730/deep_research_bench
Benchmark and evaluation pipeline for deep research agents, scoring long-form generated reports on quality with the RACE framework and on citation trustworthiness with the FACT framework, plus a public leaderboard.
TheAgentCompany/TheAgentCompany
TheAgentCompany is a benchmarking platform that evaluates large language model agents on diverse real-world professional tasks within a simulated software company environment to assess their performance and impact on work-related activities.
eleurent/rl-agents
rl-agents is a comprehensive collection of reinforcement learning and planning algorithms with tools for experiment management, benchmarking, and performance monitoring in gym-compatible environments.
LehengTHU/Agent4Rec
Agent4Rec is a recommender system simulator using 1,000 LLM-powered generative agents initialized from MovieLens-1M to simulate realistic user interactions with personalized movie recommendations.