google-deepmind/acme
Acme is a flexible and scalable research framework providing modular reinforcement learning components and agents for developing and benchmarking RL algorithms.
Awesome AI Agents › Multimodal Model Benchmarks
jylee425/b-moca
B-MoCA (Benchmarking Mobile Device Control Agents across Diverse Configurations) is a comprehensive testbed designed to evaluate and benchmark mobile device control agents across a variety of Android virtual device configurations. The project facilitates research and development in automated mobile device control by providing a standardized environment where different control agents can be tested and compared. B-MoCA supports diverse configurations by using Android emulators managed through Android Studio and integrates tools such as Android Debug Bridge (ADB) and Appium to enable interaction with the virtual devices. The environment setup involves installing Python libraries, Android Studio, Appium, and configuring Android virtual devices (AVDs) with specific snapshots that represent different device states, including variations in icon size, wallpaper, and theme. The project includes scripts and guidelines for setting up the environment, installing necessary dependencies, and managing third-party applications via APK files. It also provides automated processes for app installation and login procedures, although the latter is optional and recommended to be done with dummy accounts to protect privacy. B-MoCA supports multiple types of agents, including large language model (LLM) agents, multi-modal LLM agents, and custom agents such as those based on Llama-3 and visual language model (VLM) encoders. The framework allows for demonstration data collection, training, and evaluation of these agents, enabling researchers to benchmark their performance in zero-shot and few-shot learning scenarios. B-MoCA is particularly useful for researchers working on AI-driven mobile device automation, providing a reproducible and extensible platform to test control strategies under varied device configurations. The project emphasizes modularity and customization, allowing users to modify environment configurations and resources to suit their experimental needs. Overall, B-MoCA advances the field of mobile device control by offering a robust benchmarking suite that integrates state-of-the-art tools and methodologies for evaluating intelligent agents in realistic mobile environments.
https://github.com/jylee425/b-moca
Also in Multimodal Model Benchmarks
THUDM/AgentBench
AgentBench is a comprehensive benchmark platform designed to evaluate large language models as autonomous agents across diverse environments and tasks, facilitating research and development in LLM-based agent capabilities.
microsoft/WindowsAgentArena
Windows Agent Arena is a scalable platform for testing and benchmarking multi-modal AI agents in a realistic Windows OS environment using Azure ML for large-scale deployment.
WooooDyy/AgentGym
AgentGym is a versatile framework for developing, evaluating, and evolving large language model-based agents across diverse interactive environments with real-time feedback and scalability.
Ayanami0730/deep_research_bench
Benchmark and evaluation pipeline for deep research agents, scoring long-form generated reports on quality with the RACE framework and on citation trustworthiness with the FACT framework, plus a public leaderboard.
TheAgentCompany/TheAgentCompany
TheAgentCompany is a benchmarking platform that evaluates large language model agents on diverse real-world professional tasks within a simulated software company environment to assess their performance and impact on work-related activities.
eleurent/rl-agents
rl-agents is a comprehensive collection of reinforcement learning and planning algorithms with tools for experiment management, benchmarking, and performance monitoring in gym-compatible environments.
LehengTHU/Agent4Rec
Agent4Rec is a recommender system simulator using 1,000 LLM-powered generative agents initialized from MovieLens-1M to simulate realistic user interactions with personalized movie recommendations.