xlang-ai/OSWorld
OSWorld is a benchmarking platform for evaluating multimodal AI agents performing open-ended tasks in real computer environments using virtual machines and various virtualization technologies.
Awesome AI Agents › Sensor Fusion Agents
nottelabs/open-operator-evals
The project "open-operator-evals" by nottelabs is an open-source benchmarking framework designed to evaluate the performance of web operators or agents. It provides a transparent and reproducible benchmark that assesses how well different web agents perform across a variety of tasks on the web. The benchmark includes multiple metrics such as agent self-reported success, LLM (Large Language Model) evaluation success, time taken per task, and task reliability, which measures the percentage of tasks successfully completed at least once over multiple attempts. The project ranks various web agents, including Notte, Browser-Use, and Convergence, based on these metrics, highlighting their strengths and weaknesses. Notte leads the benchmark with the highest success rates and fastest execution times, while Browser-Use and Convergence show varying levels of performance and challenges such as CAPTCHA and bot detection. The benchmark uses a dataset called WebVoyager, which consists of approximately 600 web tasks that agents must complete, such as booking journeys or finding information on websites. To address the high variance and non-deterministic nature of web agents powered by LLMs, the project runs each task multiple times (eight runs) to average out randomness and provide a statistically sound evaluation. A smaller subset called WebVoyager30, containing 30 tasks across diverse websites, is also used to enable faster and more practical benchmarking. The project emphasizes transparency by allowing users to investigate all replays and logs and reproduce the benchmark themselves. It also provides detailed breakdowns of results for each provider, including alignment ratios and mismatch counts, which indicate how accurately agents self-assess their performance compared to the LLM evaluation. This benchmarking framework is valuable for researchers and developers working on web agents, helping them understand current capabilities, identify areas for improvement, and compare different systems objectively. Additionally, the project is actively hiring software and research engineers, indicating ongoing development and community engagement.
https://github.com/nottelabs/open-operator-evals
Also in Sensor Fusion Agents
IBM/AssetOpsBench
Open IBM framework and benchmark for building, orchestrating and evaluating domain-specific AI agents in industrial asset operations, with MCP servers over sensors, failure modes, time series and work orders.
web-arena-x/webarena
WebArena is a self-hostable web environment designed for building and evaluating autonomous agents capable of realistic web navigation and interaction tasks.
google-deepmind/android_env
AndroidEnv is a Python library by DeepMind that provides a Reinforcement Learning platform by simulating Android devices, enabling agents to interact with real-world Android applications through touchscreen gestures for diverse RL tasks.
google-research/android_world
AndroidWorld is a comprehensive environment and benchmark for autonomous agents to interact with and control Android devices through a live emulator, featuring diverse tasks and integration with web-based benchmarks.
karanchawla/GPS_IMU_Kalman_Filter
An Extended Kalman Filter implementation that fuses GPS, IMU, and encoder sensor data to accurately estimate the pose of a ground robot in a navigation frame.
OSU-NLP-Group/TravelPlanner
TravelPlanner is a benchmark for evaluating language agents in real-world travel planning tasks involving complex tool use and multiple constraints.
StonyBrookNLP/appworld
AppWorld is a high-fidelity execution environment simulating a world of apps and people to benchmark and evaluate interactive coding agents through diverse and challenging autonomous tasks.