google-deepmind/acme
Acme is a flexible and scalable research framework providing modular reinforcement learning components and agents for developing and benchmarking RL algorithms.
Awesome AI Agents › Multimodal Model Benchmarks
jylee425/mobilesafetybench
MobileSafetyBench is a comprehensive testbed designed to evaluate the safety and helpfulness of autonomous agents controlling mobile devices. It provides an environment where agents, including large language model (LLM) based agents, interact with an Android device emulator to perform various tasks. The system architecture involves a real-system device emulator, the MobileSafetyBench environment that manages interactions, and the agent executing commands. The environment maintains local variables such as goal instructions, current observations in image and text formats, action history, and task scores to facilitate task execution and evaluation. MobileSafetyBench leverages tools like Appium and Android Debug Bridge (ADB) to obtain observations and convert agent actions into device commands, enabling automated UI element manipulation and interaction. Users can customize the agent behavior by adding new agent classes or modifying prompts and observation parsing functions, allowing flexibility in experiment design. The setup requires installing Android emulators, ADB, Appium, and configuring Android Virtual Devices (AVDs) with specific system images. The project provides detailed installation guidelines for setting up the environment, including virtual environment creation, SDK installation, emulator setup, and third-party app installation via APKs. MobileSafetyBench supports multiple tasks and LLMs, with evaluation scripts and example notebooks provided to demonstrate usage. The environment also allows users to modify the action space and device settings to tailor experiments. This project is valuable for researchers and developers aiming to test and improve the safety and effectiveness of autonomous agents in mobile device control scenarios, offering a robust platform for experimentation and benchmarking.
https://github.com/jylee425/mobilesafetybench
Also in Multimodal Model Benchmarks
THUDM/AgentBench
AgentBench is a comprehensive benchmark platform designed to evaluate large language models as autonomous agents across diverse environments and tasks, facilitating research and development in LLM-based agent capabilities.
microsoft/WindowsAgentArena
Windows Agent Arena is a scalable platform for testing and benchmarking multi-modal AI agents in a realistic Windows OS environment using Azure ML for large-scale deployment.
WooooDyy/AgentGym
AgentGym is a versatile framework for developing, evaluating, and evolving large language model-based agents across diverse interactive environments with real-time feedback and scalability.
Ayanami0730/deep_research_bench
Benchmark and evaluation pipeline for deep research agents, scoring long-form generated reports on quality with the RACE framework and on citation trustworthiness with the FACT framework, plus a public leaderboard.
TheAgentCompany/TheAgentCompany
TheAgentCompany is a benchmarking platform that evaluates large language model agents on diverse real-world professional tasks within a simulated software company environment to assess their performance and impact on work-related activities.
eleurent/rl-agents
rl-agents is a comprehensive collection of reinforcement learning and planning algorithms with tools for experiment management, benchmarking, and performance monitoring in gym-compatible environments.
LehengTHU/Agent4Rec
Agent4Rec is a recommender system simulator using 1,000 LLM-powered generative agents initialized from MovieLens-1M to simulate realistic user interactions with personalized movie recommendations.