Awesome AI Agents › Browser Automation Tools

Berkeley-NLP/Agent-Eval-Refine

⭐ 149 Python added to this list on 2025-04-19 repository created 2024-04-07

The Berkeley-NLP/Agent-Eval-Refine project presents a comprehensive framework for the autonomous evaluation and refinement of digital agents that interact with web environments and mobile devices. This project is based on the research paper "Autonomous Evaluation and Refinement of Digital Agents" accepted at COLM 2024 and awarded Best Paper at the MAR Workshop of CVPR 2024. The core contribution lies in designing evaluation models that can autonomously assess and improve the performance of digital agents, which are capable of browsing the web or controlling iOS and Android devices. The repository includes evaluation models and code located in the `agent_eval` directory, which supports both open-weight models and GPT-4V-based models for assessing agent performance. It also provides tools for refining agents and running experiments on iOS and Android emulators found in the `exps` directory. Notable implementations include the Reflexion + GPT-4 agent achieving state-of-the-art results on the WebArena benchmark and a refined CogAgent model that significantly improves success rates on iOS tasks. The project offers a Python binding for mobile device emulators to facilitate end-to-end evaluation and refinement workflows. It supports evaluation of agent trajectories stored in a unified format, with utilities for inspection, annotation, and dense captioning of screenshots using a vision-language model (VLM). The captioner model enhances the modular evaluator by providing detailed descriptions of visual inputs, aiding in reasoning about agent behavior. Extensive datasets, models, and agent trajectories are publicly available on the Huggingface Hub, enabling reproducibility and further research. The repository also includes detailed instructions for setup, evaluation, and refinement processes, along with scripts and notebooks to assist users in deploying and improving digital agents across different platforms. Overall, this project advances the field of autonomous digital agents by integrating evaluation and refinement into a unified system, leveraging state-of-the-art AI models to push the boundaries of agent performance in complex interactive environments.

https://github.com/Berkeley-NLP/Agent-Eval-Refine

agent-annotationagent-captioningagent-evaluationagent-refinementagent-success-rateagent-trajectoriesandroid-emulatorautonomous-evaluationbest-paper-awardcogagentcolm-2024cvpr-2024digital-agentsfiltered-bc-refinementgpt-4huggingface-hubios-emulatormobile-device-controlpython-bindingreflexion-agentvision-language-modelvlmweb-browsing-agentswebarena