google-deepmind/acme
Acme is a flexible and scalable research framework providing modular reinforcement learning components and agents for developing and benchmarking RL algorithms.
Awesome AI Agents › Multimodal Model Benchmarks
microsoft/OpenRCA
OpenRCA is a benchmark for assessing the root cause analysis ability of large language models in software operations. Each task gives the model a natural language query about a failure and a large volume of telemetry collected from a running system, and the model must identify the time the root cause occurred, the component involved and the reason. The telemetry spans three modalities that have to be reasoned about together: KPI time series, distributed tracing graphs that expose service dependencies, and semi-structured log text. Data comes from three systems, called Telecom, Bank and Market, organised by date into log, metric and trace directories, and is distributed separately because of its size; the authors recommend at least 80 GB of disk and 32 GB of memory. The repository contains the evaluation script that scores prediction CSV files against ground truth queries, a generator that can build new tasks from a task specification file, either for the bundled telemetry or for a user''s own private telemetry, and archived predictions for reference. It also introduces RCA-agent, a baseline in which the model writes and runs Python code to retrieve and analyse the telemetry rather than reading it directly into the prompt, which keeps context length bounded and lets the approach scale to large datasets. Baselines can be reproduced with configurable API providers and models, and additional balanced and oracle baselines are included for comparison. The audience is researchers and engineers evaluating LLM agents on observability and incident diagnosis tasks. The project is Python 3.10 or newer, MIT licensed, and its FAQ documents timezone conventions, telemetry sampling limits and the rules for submitting results from custom agents.
https://github.com/microsoft/OpenRCA
Also in Multimodal Model Benchmarks
THUDM/AgentBench
AgentBench is a comprehensive benchmark platform designed to evaluate large language models as autonomous agents across diverse environments and tasks, facilitating research and development in LLM-based agent capabilities.
microsoft/WindowsAgentArena
Windows Agent Arena is a scalable platform for testing and benchmarking multi-modal AI agents in a realistic Windows OS environment using Azure ML for large-scale deployment.
WooooDyy/AgentGym
AgentGym is a versatile framework for developing, evaluating, and evolving large language model-based agents across diverse interactive environments with real-time feedback and scalability.
Ayanami0730/deep_research_bench
Benchmark and evaluation pipeline for deep research agents, scoring long-form generated reports on quality with the RACE framework and on citation trustworthiness with the FACT framework, plus a public leaderboard.
TheAgentCompany/TheAgentCompany
TheAgentCompany is a benchmarking platform that evaluates large language model agents on diverse real-world professional tasks within a simulated software company environment to assess their performance and impact on work-related activities.
eleurent/rl-agents
rl-agents is a comprehensive collection of reinforcement learning and planning algorithms with tools for experiment management, benchmarking, and performance monitoring in gym-compatible environments.
LehengTHU/Agent4Rec
Agent4Rec is a recommender system simulator using 1,000 LLM-powered generative agents initialized from MovieLens-1M to simulate realistic user interactions with personalized movie recommendations.