firecrawl/firecrawl
Firecrawl is an advanced web data API that crawls and scrapes entire websites to convert content into clean, LLM-ready markdown or structured data for AI applications.
Awesome AI Agents › Web Automation Systems
dendrite-systems/dendrite-python-sdk
Dendrite is a comprehensive framework designed to enable the creation of web AI agents that can interact with websites in a human-like manner. It provides tools for authenticating on websites, interacting with web elements, extracting structured data, downloading and uploading files, and browsing without being blocked. The SDK supports asynchronous operations and allows developers to build agents that can navigate the web, fill forms, click buttons, and extract information seamlessly. One of the key features of Dendrite is its ability to authenticate on web services by saving cookies locally after a manual login, enabling persistent sessions for AI agents. The framework supports integration with various web services, such as Outlook for sending emails, Google Analytics for extracting visitor data, and banking platforms for downloading transaction reports. Dendrite also offers functionality to convert web pages into markdown format, which is useful for feeding content into language models. The SDK is designed to be extensible and supports remote browsers hosted by Browserbase, allowing for scalable deployment of multiple AI agents in parallel without infrastructure concerns. Although the SDK is no longer under active development, it remains fully open source for learning, adaptation, and extension by the community. The project includes detailed documentation and examples to help users get started quickly and explore advanced use cases. Overall, Dendrite empowers developers to create intelligent web agents capable of complex interactions and data extraction tasks across diverse websites, enhancing automation and AI capabilities on the web.
https://github.com/dendrite-systems/dendrite-python-sdk
Also in Web Automation Systems
nanobrowser/nanobrowser
Nanobrowser is an open-source Chrome extension that enables AI-powered web automation through a multi-agent system using user-configured LLM API keys, offering a privacy-focused and cost-effective alternative to commercial tools like OpenAI Operator.
steel-dev/steel-browser
Steel Browser is an open-source browser API that enables developers to build AI-powered web agents and automation tools with full browser control, session management, proxy support, and debugging features, simplifying web automation without infrastructure overhead.
fake-useragent/fake-useragent
fake-useragent is a Python package that provides an up-to-date and customizable user-agent faker using a real-world database for realistic user-agent strings.
ishan0102/vimGPT
vimGPT is a project that combines GPT-4V's vision capabilities with the Vimium keyboard navigation extension to enable AI-assisted web browsing through visual and keyboard interactions.
brightdata/brightdata-mcp
Bright Data MCP is a powerful Model Context Protocol server that enables AI agents and applications to access and extract real-time web data seamlessly, bypassing geo-restrictions and bot protections for enhanced web scraping and navigation.
spider-rs/spider
A concurrency-first web crawler and scraper written in Rust that streams pages as they arrive and renders JavaScript only when needed, with bindings for Node.js and Python and an MCP server for agents.
nottelabs/notte
Notte is an open-source full stack framework that creates intelligent web browsing agents using a perception layer to enable fast, reliable, and cost-effective interactions with websites through large language models.