infiniflow/ragflow
RAGFlow is an open-source Retrieval-Augmented Generation engine that leverages deep document understanding and Large Language Models to provide accurate, citation-backed question-answering from complex and diverse data sources.
Awesome AI Agents › NL AI Frameworks
m92vyas/llm-reader
The project "llm-reader" is an open-source library designed to convert webpages into text that is friendly for Large Language Models (LLMs). It serves as a preprocessing tool that enhances the accuracy of data extraction and scraping from websites by transforming webpage content into a format that LLMs can easily process. This is particularly useful for applications involving Retrieval-Augmented Generation (RAG), AI web scraping, and extracting links to images and webpages, which are common tasks in web data extraction scenarios such as e-commerce scraping. The library is positioned as a fully open-source alternative to commercial APIs like Firecrawl and Jina Reader API, providing similar functionality without the associated costs. It allows users to input any webpage URL and receive LLM-ready text output, facilitating more effective and efficient data extraction workflows. While the library itself does not include anti-blocking mechanisms for web scraping, the project author recommends a paid API service called ParseExtract for users who need to scrape dynamic websites or avoid getting blocked during crawling. ParseExtract also offers additional features such as PDF, DOCX, and image parsing, OCR capabilities, and structured data and table extraction, making it a comprehensive solution for various LLM-related data extraction needs. The project includes installation instructions and example code demonstrating how to use the library to convert a URL into LLM-ready text asynchronously. It also links to additional resources, including documentation and related repositories like AI-web_scraper, which provides tools for direct scraping and web search with support for multiple LLMs. Overall, llm-reader is a valuable tool for developers and researchers working with LLMs who need to preprocess web content for improved data extraction and analysis, offering an open-source, easy-to-use solution for turning webpages into LLM-compatible input text.
https://github.com/m92vyas/llm-reader
Also in NL AI Frameworks
ScrapeGraphAI/Scrapegraph-ai
ScrapeGraphAI is an AI-powered Python library that creates intelligent web scraping pipelines using large language models to extract structured data from websites and local documents efficiently.
pydantic/pydantic
Pydantic is a Python library for fast and extensible data validation using Python type hints, enabling developers to define and validate data models efficiently.
apache/doris
Apache Doris is a high-performance, real-time analytical database with a storage-compute integrated architecture, designed for sub-second query response and high concurrency in diverse data analysis scenarios.
business-science/ai-data-science-team
AI Data Science Team is a Python library featuring AI-powered agents that automate and accelerate common data science tasks, including data cleaning, feature engineering, machine learning, and exploratory data analysis, to improve productivity and efficiency.
ucbepic/docetl
DocETL is a system for creating and executing complex document processing and data transformation pipelines powered by large language models, featuring an interactive UI playground and a Python package for production use.
hitsz-ids/synthetic-data-generator
Synthetic Data Generator (SDG) is a comprehensive framework for generating high-quality, privacy-compliant synthetic structured tabular data using advanced statistical and LLM-based models, optimized for big data applications.
Mintplex-Labs/vector-admin
VectorAdmin is a universal and user-friendly tool suite for managing multiple vector databases, providing full control over vector data with multi-user support, embedding management, and cloud deployment capabilities.