google-deepmind/acme
Acme is a flexible and scalable research framework providing modular reinforcement learning components and agents for developing and benchmarking RL algorithms.
Awesome AI Agents › Multimodal Model Benchmarks
wwh0411/FedMABench
FedMABench is an open-source benchmarking platform designed for federated training and evaluation of mobile agents in decentralized and heterogeneous user data environments. It addresses the challenges posed by data heterogeneity in federated learning by providing a comprehensive suite of datasets, algorithms, and models tailored for mobile agent applications. The benchmark includes six datasets with over 30 subsets, encompassing more than 800 applications across five categories, which simulate real-world heterogeneous data distributions such as app category distribution and specific app preferences. FedMABench supports eight federated learning algorithms including popular methods like FedAvg, FedProx, SCAFFOLD, and FedAvgM, enabling researchers to compare and evaluate different approaches under diverse conditions. The platform also offers over ten base models, featuring advanced vision-language models such as Qwen2-VL-2B/7B-Instruct, InternVL2 series, and DeepseekVL2, facilitating experimentation with state-of-the-art architectures. The benchmark is built on top of the ms-swift framework and provides detailed training and evaluation scripts to streamline the experimentation process. It supports various hardware environments including CPUs, NVIDIA RTX series GPUs, and Huawei Ascend NPUs, with optimizations for BF16 and FlashAttn on compatible devices. Users can easily install the benchmark, access datasets hosted on HuggingFace, and configure training parameters such as model type, federated learning algorithm, client number, and fine-tuning options like LoRA. FedMABench aims to foster research in federated learning for mobile agents by providing a standardized and extensible platform that captures the complexities of decentralized heterogeneous data scenarios, promoting reproducibility and benchmarking in this emerging field.
https://github.com/wwh0411/FedMABench
Also in Multimodal Model Benchmarks
THUDM/AgentBench
AgentBench is a comprehensive benchmark platform designed to evaluate large language models as autonomous agents across diverse environments and tasks, facilitating research and development in LLM-based agent capabilities.
microsoft/WindowsAgentArena
Windows Agent Arena is a scalable platform for testing and benchmarking multi-modal AI agents in a realistic Windows OS environment using Azure ML for large-scale deployment.
WooooDyy/AgentGym
AgentGym is a versatile framework for developing, evaluating, and evolving large language model-based agents across diverse interactive environments with real-time feedback and scalability.
Ayanami0730/deep_research_bench
Benchmark and evaluation pipeline for deep research agents, scoring long-form generated reports on quality with the RACE framework and on citation trustworthiness with the FACT framework, plus a public leaderboard.
TheAgentCompany/TheAgentCompany
TheAgentCompany is a benchmarking platform that evaluates large language model agents on diverse real-world professional tasks within a simulated software company environment to assess their performance and impact on work-related activities.
eleurent/rl-agents
rl-agents is a comprehensive collection of reinforcement learning and planning algorithms with tools for experiment management, benchmarking, and performance monitoring in gym-compatible environments.
LehengTHU/Agent4Rec
Agent4Rec is a recommender system simulator using 1,000 LLM-powered generative agents initialized from MovieLens-1M to simulate realistic user interactions with personalized movie recommendations.