firecrawl/firecrawl
Firecrawl is an advanced web data API that crawls and scrapes entire websites to convert content into clean, LLM-ready markdown or structured data for AI applications.
Awesome AI Agents › Web Automation Systems
McGill-NLP/weblinx
WebLINX is a comprehensive benchmark and toolkit designed for building and evaluating web navigation agents with conversational capabilities. It focuses on real-world website navigation tasks that involve multi-turn dialogues between a user and an agent. The project provides a rich dataset, WebLINX, which is accessible through Huggingface, and includes a Python library (API) to facilitate working with this dataset. The library offers abstractions such as Demonstration, Replay, and Turn to manipulate and analyze navigation demonstrations at various granularities. The repository includes code for training and evaluating state-of-the-art models reported in the associated ICML 2024 paper, such as DMR, LLaMA, MindAct, Pix2Act, and Flan-T5. These models are designed to understand and perform web navigation tasks through conversational interactions. The project also integrates with BrowserGym, a platform for web agent environments, enabling users to run agents and access demonstration steps in a standardized way. This integration is supported by the weblinx-browsergym extension and a new dataset version, WebLINX 1.1, which includes enhancements like tab actions. WebLINX provides extensive documentation, including a website, a Colab notebook for quick experimentation, and detailed API references. The evaluation module supports automatic assessment of model performance with various metrics and can be accessed both programmatically and via command line. The project encourages contributions and provides clear installation instructions for different components, including processing, video handling, evaluation, and development. Overall, WebLINX serves as a valuable resource for researchers and developers working on conversational AI agents capable of navigating complex web environments, offering tools, datasets, and models to advance this field.
https://github.com/McGill-NLP/weblinx
Also in Web Automation Systems
nanobrowser/nanobrowser
Nanobrowser is an open-source Chrome extension that enables AI-powered web automation through a multi-agent system using user-configured LLM API keys, offering a privacy-focused and cost-effective alternative to commercial tools like OpenAI Operator.
steel-dev/steel-browser
Steel Browser is an open-source browser API that enables developers to build AI-powered web agents and automation tools with full browser control, session management, proxy support, and debugging features, simplifying web automation without infrastructure overhead.
fake-useragent/fake-useragent
fake-useragent is a Python package that provides an up-to-date and customizable user-agent faker using a real-world database for realistic user-agent strings.
ishan0102/vimGPT
vimGPT is a project that combines GPT-4V's vision capabilities with the Vimium keyboard navigation extension to enable AI-assisted web browsing through visual and keyboard interactions.
brightdata/brightdata-mcp
Bright Data MCP is a powerful Model Context Protocol server that enables AI agents and applications to access and extract real-time web data seamlessly, bypassing geo-restrictions and bot protections for enhanced web scraping and navigation.
spider-rs/spider
A concurrency-first web crawler and scraper written in Rust that streams pages as they arrive and renders JavaScript only when needed, with bindings for Node.js and Python and an MCP server for agents.
nottelabs/notte
Notte is an open-source full stack framework that creates intelligent web browsing agents using a perception layer to enable fast, reliable, and cost-effective interactions with websites through large language models.