google-deepmind/acme
Acme is a flexible and scalable research framework providing modular reinforcement learning components and agents for developing and benchmarking RL algorithms.
Awesome AI Agents › Multimodal Model Benchmarks
LehengTHU/Agent4Rec
Agent4Rec is a sophisticated recommender system simulator that leverages 1,000 generative agents empowered by large language models (LLMs). These agents are initialized using the MovieLens-1M dataset, which provides a rich foundation of user preferences and social traits. Each agent simulates realistic user behavior by interacting with personalized movie recommendations in a page-by-page manner. The agents perform various actions such as watching movies, rating them, evaluating recommendations, exiting the simulation, and even interviewing, which adds depth to the simulation environment. The primary goal of Agent4Rec is to explore the potential of LLM-empowered generative agents in mimicking genuine, independent human behavior within recommendation systems. This approach allows researchers to simulate large-scale recommendation environments with a high degree of realism, which can be invaluable for testing and improving recommender algorithms. Agent4Rec supports multiple recommender system models, including Random, Popularity-based, Matrix Factorization (MF), MultVAE, and LightGCN, allowing users to experiment with different recommendation strategies. The simulation can be configured with various parameters such as the number of agents, the number of pages browsed, items per page, and execution mode (parallel or sequential), providing flexibility for diverse experimental setups. The project requires setting up a Python environment with specific dependencies, including PyTorch, and involves building extensions for accelerating recommender evaluation. It also integrates with OpenAI's API to utilize ChatGPT-3.5 for powering the generative agents, with an estimated cost for large-scale simulations. Results from simulations are saved systematically, enabling detailed analysis of agent interactions and recommendation outcomes. Overall, Agent4Rec offers a cutting-edge platform for advancing research in recommendation systems by simulating complex user-agent interactions through generative AI techniques.
https://github.com/LehengTHU/Agent4Rec
Also in Multimodal Model Benchmarks
THUDM/AgentBench
AgentBench is a comprehensive benchmark platform designed to evaluate large language models as autonomous agents across diverse environments and tasks, facilitating research and development in LLM-based agent capabilities.
microsoft/WindowsAgentArena
Windows Agent Arena is a scalable platform for testing and benchmarking multi-modal AI agents in a realistic Windows OS environment using Azure ML for large-scale deployment.
WooooDyy/AgentGym
AgentGym is a versatile framework for developing, evaluating, and evolving large language model-based agents across diverse interactive environments with real-time feedback and scalability.
Ayanami0730/deep_research_bench
Benchmark and evaluation pipeline for deep research agents, scoring long-form generated reports on quality with the RACE framework and on citation trustworthiness with the FACT framework, plus a public leaderboard.
TheAgentCompany/TheAgentCompany
TheAgentCompany is a benchmarking platform that evaluates large language model agents on diverse real-world professional tasks within a simulated software company environment to assess their performance and impact on work-related activities.
eleurent/rl-agents
rl-agents is a comprehensive collection of reinforcement learning and planning algorithms with tools for experiment management, benchmarking, and performance monitoring in gym-compatible environments.
web-arena-x/visualwebarena
VisualWebArena is a benchmark for evaluating autonomous multimodal language agents on complex and realistic web-based visual tasks.