Awesome AI Agents › Web Agents
Tarsier is a sophisticated vision utility designed to enhance web interaction agents by providing a powerful perception system for understanding and interacting with web pages. It addresses common challenges faced when using large language models (LLMs) to automate web interactions, such as how to represent a webpage to an LLM, how to map LLM responses back to specific web elements, and how to convey the visual structure of a page to text-only LLMs. The core functionality of Tarsier involves visually tagging interactable elements on a webpage with unique identifiers, enabling LLMs to perform actions like clicking or typing on these elements through simple commands referencing these tags. Interactable elements include buttons, links, and input fields, with an option to tag all textual elements as well. A key innovation in Tarsier is its OCR algorithm that converts webpage screenshots into whitespace-structured text representations, akin to ASCII art. This allows LLMs without vision capabilities to understand the page layout and content effectively. Benchmark results indicate that unimodal GPT-4 combined with Tarsier's text representation outperforms multimodal GPT-4 with screenshot input by 10-20%, highlighting the effectiveness of this approach. Tarsier supports integration with popular OCR services like Google Cloud Vision and Microsoft Azure Computer Vision, facilitating accurate text extraction from images. The project provides comprehensive documentation, including example notebooks demonstrating autonomous web agents built with LangChain and LlamaIndex frameworks. It also offers tools for local development, testing, and linting to support contributors. Overall, Tarsier is a valuable tool for developers building intelligent web automation agents, enabling more precise and visually informed interactions with web content through LLMs. Its combination of element tagging and advanced OCR techniques makes it a unique solution in the space of web interaction automation.
https://github.com/reworkd/tarsier
LaVague is an open-source Large Action Model framework for developing AI Web Agents that automate web-based tasks using components like a World Model and Action Engine with support for Selenium, Playwright, and Chrome extension drivers.
Index is an advanced open-source browser agent that autonomously performs complex web tasks using AI models, offering hosted and self-hosted options with strong observability features.
A curated list of tools, frameworks, and resources for building AI web agents that autonomously browse and interact with the web.
WebVoyager is an end-to-end web agent powered by Large Multimodal Models that interacts with real-world websites to complete diverse user tasks using textual and visual information.
Python SDK for the Oxylabs AI Studio scrape agent, which extracts data from a single web page using a natural language prompt and returns structured JSON or Markdown without CSS or XPath selectors.
AgentLab is an open-source framework for developing, testing, and benchmarking scalable and reproducible web agents across diverse tasks using BrowserGym benchmarks and large-scale parallel experiments.
WebShop is a simulated e-commerce environment designed to train and evaluate grounded language agents for complex real-world web interaction tasks involving product search and purchase based on natural language instructions.
Ash Agents is a Python swarm framework that performs web tasks through disposable agents: each runs in an isolated browser context with its own fingerprint and egress, then is destroyed so no interaction links back to the user.