firecrawl/firecrawl
Firecrawl is an advanced web data API that crawls and scrapes entire websites to convert content into clean, LLM-ready markdown or structured data for AI applications.
Awesome AI Agents › Web Automation Systems
theredsix/cerebellum
Cerebellum is a lightweight browser automation system designed to accomplish user-defined goals on web pages by simulating keyboard and mouse actions. It operates by simplifying web browsing into navigating a directed graph where each webpage represents a node containing visible elements and data, and user actions such as clicking or typing represent edges that transition between these nodes. The system starts at an initial webpage and aims to reach a target node that signifies the completion of the user's goal. It leverages a large language model (LLM), specifically Claude 3.5 Sonnet, to analyze page content and interactive elements to discover new nodes and decide the next action based on the current state and past actions. This decision-making process is iterative, with Cerebellum executing the LLM's planned actions and feeding the resulting new state back into the LLM until the goal is achieved or deemed unachievable. Cerebellum is compatible with any Selenium-supported browser and supports form filling using user-provided JSON data. It also accepts runtime instructions to dynamically adjust browsing strategies and actions. The project is actively being developed with a roadmap that includes handling tabbed browsing, improving scrolling behaviors, adding data extraction capabilities, supporting additional LLMs, and creating training datasets from browsing sessions. Known issues include limitations with the current LLM's refusal to solve CAPTCHAs and navigate pages with political content. The project is open for contributions and is maintained by Han Wang along with several collaborators. It is licensed under the MIT License. Cerebellum offers a novel approach to browser automation by integrating AI-driven planning to navigate complex web interactions, making it a powerful tool for automating web-based tasks that require goal-oriented browsing and interaction.
https://github.com/theredsix/cerebellum
Also in Web Automation Systems
nanobrowser/nanobrowser
Nanobrowser is an open-source Chrome extension that enables AI-powered web automation through a multi-agent system using user-configured LLM API keys, offering a privacy-focused and cost-effective alternative to commercial tools like OpenAI Operator.
steel-dev/steel-browser
Steel Browser is an open-source browser API that enables developers to build AI-powered web agents and automation tools with full browser control, session management, proxy support, and debugging features, simplifying web automation without infrastructure overhead.
fake-useragent/fake-useragent
fake-useragent is a Python package that provides an up-to-date and customizable user-agent faker using a real-world database for realistic user-agent strings.
ishan0102/vimGPT
vimGPT is a project that combines GPT-4V's vision capabilities with the Vimium keyboard navigation extension to enable AI-assisted web browsing through visual and keyboard interactions.
brightdata/brightdata-mcp
Bright Data MCP is a powerful Model Context Protocol server that enables AI agents and applications to access and extract real-time web data seamlessly, bypassing geo-restrictions and bot protections for enhanced web scraping and navigation.
spider-rs/spider
A concurrency-first web crawler and scraper written in Rust that streams pages as they arrive and renders JavaScript only when needed, with bindings for Node.js and Python and an MCP server for agents.
nottelabs/notte
Notte is an open-source full stack framework that creates intelligent web browsing agents using a perception layer to enable fast, reliable, and cost-effective interactions with websites through large language models.