firecrawl/firecrawl
Firecrawl is an advanced web data API that crawls and scrapes entire websites to convert content into clean, LLM-ready markdown or structured data for AI applications.
Awesome AI Agents › Web Automation Systems
steel-dev/surf.new
surf.new is an open-source project that serves as a playground for testing web agents powered by Steel.dev. It allows users to experience how different AI agents can surf and interact with the web in a human-like manner. The platform supports multiple AI providers including Claude and GPT-4, enabling diverse AI agent interactions. It features real-time browser interaction visualization, providing users with a clear view of how agents operate within a controlled environment. The project is built using modern web technologies such as Next.js 15 for the frontend, FastAPI for the backend, and the Steel SDK for browser automation. The user interface is designed with Shadcn UI components and styled using Tailwind CSS, ensuring a beautiful and responsive experience. Additionally, the project integrates the Vercel AI SDK for AI chat interfaces and Langchain as an agent framework, enhancing its AI capabilities. Users can easily set up the project locally by cloning the repository, installing dependencies for both frontend and backend, and configuring environment variables including a Steel API key. The project also provides specific instructions for Windows users to handle asyncio event loop differences to ensure smooth operation with Playwright for browser automation. surf.new encourages community contributions and provides clear guidelines for forking, branching, committing, and submitting pull requests. The project is licensed under the MIT License and is actively maintained by the Steel team. It also offers links to the main Steel.dev platform, documentation, Discord community, and Twitter for further engagement and support. Overall, surf.new is a comprehensive and user-friendly platform for experimenting with AI-driven web agents, making it a valuable tool for developers and researchers interested in AI and web automation.
https://github.com/steel-dev/surf.new
Also in Web Automation Systems
nanobrowser/nanobrowser
Nanobrowser is an open-source Chrome extension that enables AI-powered web automation through a multi-agent system using user-configured LLM API keys, offering a privacy-focused and cost-effective alternative to commercial tools like OpenAI Operator.
steel-dev/steel-browser
Steel Browser is an open-source browser API that enables developers to build AI-powered web agents and automation tools with full browser control, session management, proxy support, and debugging features, simplifying web automation without infrastructure overhead.
fake-useragent/fake-useragent
fake-useragent is a Python package that provides an up-to-date and customizable user-agent faker using a real-world database for realistic user-agent strings.
ishan0102/vimGPT
vimGPT is a project that combines GPT-4V's vision capabilities with the Vimium keyboard navigation extension to enable AI-assisted web browsing through visual and keyboard interactions.
brightdata/brightdata-mcp
Bright Data MCP is a powerful Model Context Protocol server that enables AI agents and applications to access and extract real-time web data seamlessly, bypassing geo-restrictions and bot protections for enhanced web scraping and navigation.
spider-rs/spider
A concurrency-first web crawler and scraper written in Rust that streams pages as they arrive and renders JavaScript only when needed, with bindings for Node.js and Python and an MCP server for agents.
nottelabs/notte
Notte is an open-source full stack framework that creates intelligent web browsing agents using a perception layer to enable fast, reliable, and cost-effective interactions with websites through large language models.