firecrawl/firecrawl
Firecrawl is an advanced web data API that crawls and scrapes entire websites to convert content into clean, LLM-ready markdown or structured data for AI applications.
Awesome AI Agents › Web Automation Systems
iMeanAI/WebCanvas
WebCanvas is an all-in-one open-source framework designed for building, training, and evaluating web agents in live, dynamic web environments. Unlike existing frameworks that are either offline or operate in static, reproducible environments, WebCanvas pioneers the online development and assessment of web agents interacting with real-time web content. The framework provides a comprehensive suite of toolkits that support the scaling and maintenance of key-node based web trajectories, enabling detailed and phase-based evaluation of agent performance. Key features include a modular base agent framework with plug-and-play components such as Planning, Observation, Memory, Reward, Action Execution, and Evaluation, which facilitate easy development and iteration of LLM-based web agents. WebCanvas emphasizes realistic interaction by leveraging live web environments, addressing challenges like error handling, CAPTCHA solving, and efficiency metrics. The introduction of "key nodes" allows for granular feedback and rigorous assessment during task execution. The framework supports a broad action space and accurate observation capture, enabling robust evaluation of open-ended web tasks. It integrates with cloud browser environments for stable testing and supports multiple base model services including OpenAI, Claude, Gemini, and together.ai. The project also offers the Mind2Web-Live dataset, a refined benchmark containing over 500 tasks with thousands of intermediate evaluation states, along with open access to raw data such as HTML, screenshots, DOM trees, and operation videos for research purposes. WebCanvas is actively maintained with frequent updates, including new evaluation systems, metrics for agent efficiency, and integrations with other agent frameworks. The roadmap includes plans for enhanced modularity, dynamic evaluation functions, expanded dataset coverage, improved error handling algorithms, and better visualization tools to track agent performance. This framework is ideal for researchers and developers aiming to advance the capabilities of web agents in realistic online settings.
https://github.com/iMeanAI/WebCanvas
Also in Web Automation Systems
nanobrowser/nanobrowser
Nanobrowser is an open-source Chrome extension that enables AI-powered web automation through a multi-agent system using user-configured LLM API keys, offering a privacy-focused and cost-effective alternative to commercial tools like OpenAI Operator.
steel-dev/steel-browser
Steel Browser is an open-source browser API that enables developers to build AI-powered web agents and automation tools with full browser control, session management, proxy support, and debugging features, simplifying web automation without infrastructure overhead.
fake-useragent/fake-useragent
fake-useragent is a Python package that provides an up-to-date and customizable user-agent faker using a real-world database for realistic user-agent strings.
ishan0102/vimGPT
vimGPT is a project that combines GPT-4V's vision capabilities with the Vimium keyboard navigation extension to enable AI-assisted web browsing through visual and keyboard interactions.
brightdata/brightdata-mcp
Bright Data MCP is a powerful Model Context Protocol server that enables AI agents and applications to access and extract real-time web data seamlessly, bypassing geo-restrictions and bot protections for enhanced web scraping and navigation.
spider-rs/spider
A concurrency-first web crawler and scraper written in Rust that streams pages as they arrive and renders JavaScript only when needed, with bindings for Node.js and Python and an MCP server for agents.
nottelabs/notte
Notte is an open-source full stack framework that creates intelligent web browsing agents using a perception layer to enable fast, reliable, and cost-effective interactions with websites through large language models.