firecrawl/firecrawl
Firecrawl is an advanced web data API that crawls and scrapes entire websites to convert content into clean, LLM-ready markdown or structured data for AI applications.
Awesome AI Agents › Web Automation Systems
fake-useragent/fake-useragent
The project "fake-useragent" is a Python package designed to provide an up-to-date and simple user-agent faker using a real-world database. It offers a comprehensive collection of user-agent strings from various browsers and platforms, allowing developers to simulate different user environments for testing or scraping purposes. The data is pre-downloaded and post-processed from Intoli LLC's user-agents repository, ensuring a wide range of browser agents and types including desktop, tablet, and mobile user agents. Users can retrieve user-agent strings or detailed Python dictionaries containing fields such as useragent, percent, type, device_brand, browser, browser_version, OS, OS_version, and platform. The package supports Python 3.9 and higher. The library provides simple usage with the ability to get random user-agent strings or specific browser user-agents like Chrome, Firefox, Safari, Opera, Edge, and more. It also allows customization by specifying browsers, operating systems, and platform types to filter the user-agent results. Additional features include setting minimum browser version requirements and combining multiple filters for more precise user-agent selection. Since version 1.3.0, the package offers access to the full user-agent dictionary for more detailed information. It also includes fallback mechanisms to handle rare cases where user-agent retrieval might fail. The project is actively maintained with continuous integration workflows for testing, linting, and security analysis. Overall, fake-useragent is a valuable tool for developers needing realistic user-agent strings for web scraping, automated testing, or any scenario requiring user-agent spoofing. Its extensive database, ease of use, and customization options make it a robust solution for simulating diverse browsing environments.
https://github.com/fake-useragent/fake-useragent
Also in Web Automation Systems
nanobrowser/nanobrowser
Nanobrowser is an open-source Chrome extension that enables AI-powered web automation through a multi-agent system using user-configured LLM API keys, offering a privacy-focused and cost-effective alternative to commercial tools like OpenAI Operator.
steel-dev/steel-browser
Steel Browser is an open-source browser API that enables developers to build AI-powered web agents and automation tools with full browser control, session management, proxy support, and debugging features, simplifying web automation without infrastructure overhead.
ishan0102/vimGPT
vimGPT is a project that combines GPT-4V's vision capabilities with the Vimium keyboard navigation extension to enable AI-assisted web browsing through visual and keyboard interactions.
brightdata/brightdata-mcp
Bright Data MCP is a powerful Model Context Protocol server that enables AI agents and applications to access and extract real-time web data seamlessly, bypassing geo-restrictions and bot protections for enhanced web scraping and navigation.
spider-rs/spider
A concurrency-first web crawler and scraper written in Rust that streams pages as they arrive and renders JavaScript only when needed, with bindings for Node.js and Python and an MCP server for agents.
nottelabs/notte
Notte is an open-source full stack framework that creates intelligent web browsing agents using a perception layer to enable fast, reliable, and cost-effective interactions with websites through large language models.
TaxyAI/browser-extension
Taxy AI is an open-source Chrome extension that uses GPT-4 to automate repetitive browser tasks by interpreting natural language instructions and interacting with web pages through an action cycle.