firecrawl/firecrawl
Firecrawl is an advanced web data API that crawls and scrapes entire websites to convert content into clean, LLM-ready markdown or structured data for AI applications.
Awesome AI Agents › Web Automation Systems
brightdata/brightdata-mcp
Bright Data MCP is a robust Model Context Protocol (MCP) server designed to empower AI agents, large language models (LLMs), and applications with real-time access to web data. It serves as an all-in-one solution for public web access, enabling seamless web searching, navigation, data extraction, and interaction without the risk of being blocked, making it ideal for web scraping tasks. The server supports integration with various MCP-compatible AI assistants such as Claude Desktop, Cursor, and Windsurf, enhancing their capabilities by providing up-to-date information directly from the web. Key features include real-time web access, bypassing geo-restrictions to access content regardless of location, and a Web Unlocker that helps navigate websites protected by bot detection mechanisms. Additionally, it offers optional remote browser automation for more advanced web interactions. The server is designed with security best practices in mind, advising users to treat scraped content as untrusted data and to use structured data extraction to avoid prompt injection risks. Setting up the Bright Data MCP involves creating an account on brightdata.com, obtaining an API key, and optionally configuring custom Web Unlocker and Browser API zones for enhanced control. The server can be run using the command `npx @brightdata/mcp` with the necessary environment variables set. It also provides detailed documentation, usage examples, and troubleshooting tips to assist users in deployment and operation. The project includes demos and playgrounds for users to experiment with its capabilities without local setup. It is actively maintained with a changelog and welcomes community contributions through issue reporting and pull requests. Overall, Bright Data MCP is a powerful tool for integrating real-time web data into AI workflows, enabling smarter, context-aware applications and agents.
https://github.com/brightdata/brightdata-mcp
Also in Web Automation Systems
nanobrowser/nanobrowser
Nanobrowser is an open-source Chrome extension that enables AI-powered web automation through a multi-agent system using user-configured LLM API keys, offering a privacy-focused and cost-effective alternative to commercial tools like OpenAI Operator.
steel-dev/steel-browser
Steel Browser is an open-source browser API that enables developers to build AI-powered web agents and automation tools with full browser control, session management, proxy support, and debugging features, simplifying web automation without infrastructure overhead.
fake-useragent/fake-useragent
fake-useragent is a Python package that provides an up-to-date and customizable user-agent faker using a real-world database for realistic user-agent strings.
ishan0102/vimGPT
vimGPT is a project that combines GPT-4V's vision capabilities with the Vimium keyboard navigation extension to enable AI-assisted web browsing through visual and keyboard interactions.
spider-rs/spider
A concurrency-first web crawler and scraper written in Rust that streams pages as they arrive and renders JavaScript only when needed, with bindings for Node.js and Python and an MCP server for agents.
nottelabs/notte
Notte is an open-source full stack framework that creates intelligent web browsing agents using a perception layer to enable fast, reliable, and cost-effective interactions with websites through large language models.
TaxyAI/browser-extension
Taxy AI is an open-source Chrome extension that uses GPT-4 to automate repetitive browser tasks by interpreting natural language instructions and interacting with web pages through an action cycle.