firecrawl/firecrawl
Firecrawl is an advanced web data API that crawls and scrapes entire websites to convert content into clean, LLM-ready markdown or structured data for AI applications.
Awesome AI Agents › Web Automation Systems
nottelabs/notte
Notte is an open-source full stack framework designed to create intelligent web browsing agents powered by large language models (LLMs). It introduces a unique perception layer that transforms websites into structured, natural language maps, making it easier for LLMs to understand and interact with web content efficiently. This abstraction reduces the cognitive load on the reasoning models, enabling the use of smaller, faster models while maintaining high accuracy and reliability. Notte outperforms other web agents in speed, cost, and reliability, as demonstrated by benchmarks where it ranks first in task reliability and speed compared to similar projects. The framework supports a wide range of functionalities including session management, live replay, browser control via Chrome DevTools Protocol (CDP), web scraping, and automated web interactions. It offers a comprehensive API with endpoints for scraping, session handling, page observation, and agent task execution. Notte also provides a hosted SDK for managing cloud browser sessions and running agents remotely, making it accessible for developers to integrate into their applications. Notte's architecture is highly customizable and designed for production deployment, allowing integration with other developer tools and ecosystems. It supports advanced features like secure password vaults, bot detection bypass, and is working on upcoming capabilities such as captcha resolution, residential proxies, VPN-style browsing, authentication, payments, and human-in-the-loop integration. The project emphasizes speed and efficiency, leveraging the perception layer to enable ultra-high inference speeds on specialized hardware like Cerebras without sacrificing precision. It is actively maintained and documented, with a strong community presence and continuous improvements. Notte represents a significant step towards agentic internet layers and is considered one of the closest attempts to artificial general intelligence (AGI) in web browsing agents.
https://github.com/nottelabs/notte
Also in Web Automation Systems
nanobrowser/nanobrowser
Nanobrowser is an open-source Chrome extension that enables AI-powered web automation through a multi-agent system using user-configured LLM API keys, offering a privacy-focused and cost-effective alternative to commercial tools like OpenAI Operator.
steel-dev/steel-browser
Steel Browser is an open-source browser API that enables developers to build AI-powered web agents and automation tools with full browser control, session management, proxy support, and debugging features, simplifying web automation without infrastructure overhead.
fake-useragent/fake-useragent
fake-useragent is a Python package that provides an up-to-date and customizable user-agent faker using a real-world database for realistic user-agent strings.
ishan0102/vimGPT
vimGPT is a project that combines GPT-4V's vision capabilities with the Vimium keyboard navigation extension to enable AI-assisted web browsing through visual and keyboard interactions.
brightdata/brightdata-mcp
Bright Data MCP is a powerful Model Context Protocol server that enables AI agents and applications to access and extract real-time web data seamlessly, bypassing geo-restrictions and bot protections for enhanced web scraping and navigation.
spider-rs/spider
A concurrency-first web crawler and scraper written in Rust that streams pages as they arrive and renders JavaScript only when needed, with bindings for Node.js and Python and an MCP server for agents.
TaxyAI/browser-extension
Taxy AI is an open-source Chrome extension that uses GPT-4 to automate repetitive browser tasks by interpreting natural language instructions and interacting with web pages through an action cycle.