firecrawl/firecrawl
Firecrawl is an advanced web data API that crawls and scrapes entire websites to convert content into clean, LLM-ready markdown or structured data for AI applications.
Awesome AI Agents › Web Automation Systems
OTA-Tech-AI/web-agent-protocol
The Web Agent Protocol (WAP) is a comprehensive and standardized framework designed to facilitate seamless interaction between users, web agents, and browsers by recording and replaying user interactions within the browser environment. It provides a structured approach to capturing user actions, converting these actions into reusable formats, and replaying them accurately to automate browser operations. The protocol distinctly separates the processes of action recording and execution, enhancing automation efficiency and reusability. WAP includes a Python SDK that fully implements the protocol specification, making it accessible and easy to use for developers. The system works in conjunction with the OTA-WAP Chrome extension, which captures user interaction data directly from the browser. This data is then processed into two types of replay lists: exact-replay, which reproduces every recorded action precisely, and smart-replay, which condenses actions into goal-oriented steps for more efficient automation. Additionally, WAP supports converting recorded actions into MCP (Model Context Protocol) servers, enabling reuse by any agent or user. The replay functionality is powered by the WAP-Replay protocol, ensuring that browser operations are executed accurately and reliably. The project also provides tools for setting up a data collection server, generating replay lists, and running replays either through command-line interfaces or a dedicated desktop application. The framework is designed to be flexible and extensible, supporting various modes of replay and integration with external APIs such as OpenAI and DeepSeek. It includes detailed setup instructions, troubleshooting tips, and examples to help users get started quickly. The project is open-source under the MIT license and acknowledges contributions from related projects like Browser-Use, MCP SDK, and DOMListenerExtension. Overall, WAP aims to streamline browser automation by providing a robust protocol and toolset for recording, converting, and replaying user interactions, making it valuable for developers working on web automation, testing, and AI-driven web agents.
https://github.com/OTA-Tech-AI/web-agent-protocol
Also in Web Automation Systems
nanobrowser/nanobrowser
Nanobrowser is an open-source Chrome extension that enables AI-powered web automation through a multi-agent system using user-configured LLM API keys, offering a privacy-focused and cost-effective alternative to commercial tools like OpenAI Operator.
steel-dev/steel-browser
Steel Browser is an open-source browser API that enables developers to build AI-powered web agents and automation tools with full browser control, session management, proxy support, and debugging features, simplifying web automation without infrastructure overhead.
fake-useragent/fake-useragent
fake-useragent is a Python package that provides an up-to-date and customizable user-agent faker using a real-world database for realistic user-agent strings.
ishan0102/vimGPT
vimGPT is a project that combines GPT-4V's vision capabilities with the Vimium keyboard navigation extension to enable AI-assisted web browsing through visual and keyboard interactions.
brightdata/brightdata-mcp
Bright Data MCP is a powerful Model Context Protocol server that enables AI agents and applications to access and extract real-time web data seamlessly, bypassing geo-restrictions and bot protections for enhanced web scraping and navigation.
spider-rs/spider
A concurrency-first web crawler and scraper written in Rust that streams pages as they arrive and renders JavaScript only when needed, with bindings for Node.js and Python and an MCP server for agents.
nottelabs/notte
Notte is an open-source full stack framework that creates intelligent web browsing agents using a perception layer to enable fast, reliable, and cost-effective interactions with websites through large language models.