firecrawl/firecrawl
Firecrawl is an advanced web data API that crawls and scrapes entire websites to convert content into clean, LLM-ready markdown or structured data for AI applications.
Awesome AI Agents › Web Automation Systems
steel-dev/steel-browser
Steel Browser is an open-source browser API designed specifically for AI agents and applications, providing a comprehensive and ready-to-use browser instance that simplifies web automation without the need to manage complex infrastructure. It enables developers to build live web agents and browser automation tools efficiently by handling the underlying browser processes, sessions, and pages. The API offers full control over Chrome instances through Puppeteer and Chrome DevTools Protocol (CDP), allowing integration with popular automation frameworks like Puppeteer, Playwright, and Selenium. Key features include session management that preserves browser state, cookies, and local storage across requests, built-in proxy chain management for IP rotation, and support for loading custom Chrome extensions to enhance functionality. The platform also incorporates anti-detection mechanisms such as stealth plugins and fingerprint management to avoid detection during automated browsing. Developers benefit from built-in debugging tools including request logging and a user interface for session inspection and troubleshooting. Steel Browser manages browser lifecycle and resource cleanup automatically, ensuring efficient resource use. It also provides APIs for converting web pages into markdown, readability formats, screenshots, or PDFs, facilitating content extraction and processing. The project supports deployment via Docker, cloud providers, or local environments, with detailed documentation and examples available to help users get started quickly. The project is actively maintained and encourages community involvement through GitHub and Discord. It is suitable for developers looking to build AI-powered web agents, scraping tools, or browser automation solutions without the overhead of managing browser infrastructure. The availability of SDKs for Node.js and Python further simplifies integration into various development workflows.
https://github.com/steel-dev/steel-browser
Also in Web Automation Systems
nanobrowser/nanobrowser
Nanobrowser is an open-source Chrome extension that enables AI-powered web automation through a multi-agent system using user-configured LLM API keys, offering a privacy-focused and cost-effective alternative to commercial tools like OpenAI Operator.
fake-useragent/fake-useragent
fake-useragent is a Python package that provides an up-to-date and customizable user-agent faker using a real-world database for realistic user-agent strings.
ishan0102/vimGPT
vimGPT is a project that combines GPT-4V's vision capabilities with the Vimium keyboard navigation extension to enable AI-assisted web browsing through visual and keyboard interactions.
brightdata/brightdata-mcp
Bright Data MCP is a powerful Model Context Protocol server that enables AI agents and applications to access and extract real-time web data seamlessly, bypassing geo-restrictions and bot protections for enhanced web scraping and navigation.
spider-rs/spider
A concurrency-first web crawler and scraper written in Rust that streams pages as they arrive and renders JavaScript only when needed, with bindings for Node.js and Python and an MCP server for agents.
nottelabs/notte
Notte is an open-source full stack framework that creates intelligent web browsing agents using a perception layer to enable fast, reliable, and cost-effective interactions with websites through large language models.
TaxyAI/browser-extension
Taxy AI is an open-source Chrome extension that uses GPT-4 to automate repetitive browser tasks by interpreting natural language instructions and interacting with web pages through an action cycle.