firecrawl/firecrawl
Firecrawl is an advanced web data API that crawls and scrapes entire websites to convert content into clean, LLM-ready markdown or structured data for AI applications.
Awesome AI Agents › Web Automation Systems
spider-rs/spider
Spider is a crawling and scraping engine written in Rust and built around concurrency. It fetches over plain HTTP first and launches headless Chrome only for pages that actually require JavaScript, a strategy exposed through the crawl_smart call when the chrome feature is enabled. Streaming is built into both the HTTP and Chrome paths, so pages are delivered to the caller as they are fetched instead of being batched at the end; the typical usage subscribes to a channel on a Website value and receives each page with its status code and URL while the crawler discovers links, respects configured boundaries and terminates on its own. Configuration options all have defaults and cover the concurrent request limit, crawl depth, delay between requests, respect for robots.txt, whether subdomains are followed, the user agent string and a stealth mode; proxies, retries and rate limiting are part of the engine. The same code scales from a single asynchronous task to a distributed worker fleet without changes, and a configuration switch routes traffic through the commercial Spider Cloud service, whose Smart mode proxies by default and escalates to an unblocker only on pages that resist. Distribution covers several ecosystems: a Rust crate, a command-line tool, a Node.js package, a Python package and an MCP server crate that exposes crawling to agent clients such as Claude and Cursor. The README describes typical uses as feeding web content into vector stores for LLM and retrieval-augmented generation pipelines, monitoring sites for search-engine and price changes, exporting pages as Markdown, JSON or WARC, and driving headless Chrome for AI browsing agents, and the repository ships more than fifty runnable examples. The project is MIT licensed.
https://github.com/spider-rs/spider
Also in Web Automation Systems
nanobrowser/nanobrowser
Nanobrowser is an open-source Chrome extension that enables AI-powered web automation through a multi-agent system using user-configured LLM API keys, offering a privacy-focused and cost-effective alternative to commercial tools like OpenAI Operator.
steel-dev/steel-browser
Steel Browser is an open-source browser API that enables developers to build AI-powered web agents and automation tools with full browser control, session management, proxy support, and debugging features, simplifying web automation without infrastructure overhead.
fake-useragent/fake-useragent
fake-useragent is a Python package that provides an up-to-date and customizable user-agent faker using a real-world database for realistic user-agent strings.
ishan0102/vimGPT
vimGPT is a project that combines GPT-4V's vision capabilities with the Vimium keyboard navigation extension to enable AI-assisted web browsing through visual and keyboard interactions.
brightdata/brightdata-mcp
Bright Data MCP is a powerful Model Context Protocol server that enables AI agents and applications to access and extract real-time web data seamlessly, bypassing geo-restrictions and bot protections for enhanced web scraping and navigation.
nottelabs/notte
Notte is an open-source full stack framework that creates intelligent web browsing agents using a perception layer to enable fast, reliable, and cost-effective interactions with websites through large language models.
TaxyAI/browser-extension
Taxy AI is an open-source Chrome extension that uses GPT-4 to automate repetitive browser tasks by interpreting natural language instructions and interacting with web pages through an action cycle.