firecrawl/firecrawl
Firecrawl is an advanced web data API that crawls and scrapes entire websites to convert content into clean, LLM-ready markdown or structured data for AI applications.
Awesome AI Agents › Web Automation Systems
kohjingyu/search-agents
The project "Tree Search for Language Model Agents" presents an inference-time tree search algorithm designed to enhance language model agents' capabilities in exploration and multi-step planning within interactive web environments. This approach is particularly demonstrated on the VisualWebArena and WebArena benchmarks, which are platforms for testing agent performance in web-based tasks. The repository provides comprehensive instructions for installation, environment setup, and running evaluations, including configuring URLs for various websites and setting up API keys for OpenAI models. It supports running experiments with different agents, including GPT-4o and Llama-3 models, with detailed scripts to reproduce results from the associated research paper. The project also releases agent trajectories and results, facilitating further research and validation. The codebase builds upon existing VisualWebArena and WebArena projects, integrating advanced tree search techniques to improve agent decision-making and planning in complex web environments. This work is significant for advancing the capabilities of language model agents in dynamic and interactive settings, enabling more sophisticated and effective task completion through strategic exploration and planning.
https://github.com/kohjingyu/search-agents
Also in Web Automation Systems
nanobrowser/nanobrowser
Nanobrowser is an open-source Chrome extension that enables AI-powered web automation through a multi-agent system using user-configured LLM API keys, offering a privacy-focused and cost-effective alternative to commercial tools like OpenAI Operator.
steel-dev/steel-browser
Steel Browser is an open-source browser API that enables developers to build AI-powered web agents and automation tools with full browser control, session management, proxy support, and debugging features, simplifying web automation without infrastructure overhead.
fake-useragent/fake-useragent
fake-useragent is a Python package that provides an up-to-date and customizable user-agent faker using a real-world database for realistic user-agent strings.
ishan0102/vimGPT
vimGPT is a project that combines GPT-4V's vision capabilities with the Vimium keyboard navigation extension to enable AI-assisted web browsing through visual and keyboard interactions.
brightdata/brightdata-mcp
Bright Data MCP is a powerful Model Context Protocol server that enables AI agents and applications to access and extract real-time web data seamlessly, bypassing geo-restrictions and bot protections for enhanced web scraping and navigation.
spider-rs/spider
A concurrency-first web crawler and scraper written in Rust that streams pages as they arrive and renders JavaScript only when needed, with bindings for Node.js and Python and an MCP server for agents.
nottelabs/notte
Notte is an open-source full stack framework that creates intelligent web browsing agents using a perception layer to enable fast, reliable, and cost-effective interactions with websites through large language models.