infiniflow/ragflow
RAGFlow is an open-source Retrieval-Augmented Generation engine that leverages deep document understanding and Large Language Models to provide accurate, citation-backed question-answering from complex and diverse data sources.
Awesome AI Agents › NL AI Frameworks
bespokelabsai/curator
Bespoke Curator is a powerful Python-based library designed for synthetic data curation, particularly useful for post-training and structured data extraction tasks. It facilitates the creation of synthetic data pipelines that enable users to generate and curate high-quality data efficiently and robustly. The project supports bulk inference and scalable data curation, making it suitable for large-scale data preparation needs. The library offers rich features including a viewer to monitor data generation in real-time, first-class support for structured outputs, and built-in performance optimizations such as asynchronous operations, caching, and fault recovery to ensure reliability at any scale. It supports a wide range of inference options through integrations with LiteLLM, vLLM, and popular batch APIs, enhancing its flexibility and usability across different environments. Bespoke Curator has been actively used to create notable datasets like OpenThoughts2-1M and Bespoke-Stratos-17k, which are high-quality reasoning datasets trending on platforms like HuggingFace. It also supports advanced functionalities such as batch processing for APIs like OpenAI and Anthropic, code execution with multiple backend options (local multiprocessing, Ray, Docker, e2b), and multimodal data generation. The project provides extensive documentation, tutorials, and guides to help users get started and make the most of its capabilities. It includes practical examples for tasks such as product feature extraction, sentiment analysis, domain-specific retrieval augmented fine-tuning (RAFT), reasoning dataset generation, ungrounded question-answer generation, and synthetic chart creation. Overall, Bespoke Curator is a comprehensive tool for researchers and developers looking to streamline synthetic data generation and curation workflows, improve model training datasets, and extract structured data efficiently with robust performance and scalability.
https://github.com/bespokelabsai/curator
Also in NL AI Frameworks
ScrapeGraphAI/Scrapegraph-ai
ScrapeGraphAI is an AI-powered Python library that creates intelligent web scraping pipelines using large language models to extract structured data from websites and local documents efficiently.
pydantic/pydantic
Pydantic is a Python library for fast and extensible data validation using Python type hints, enabling developers to define and validate data models efficiently.
apache/doris
Apache Doris is a high-performance, real-time analytical database with a storage-compute integrated architecture, designed for sub-second query response and high concurrency in diverse data analysis scenarios.
business-science/ai-data-science-team
AI Data Science Team is a Python library featuring AI-powered agents that automate and accelerate common data science tasks, including data cleaning, feature engineering, machine learning, and exploratory data analysis, to improve productivity and efficiency.
ucbepic/docetl
DocETL is a system for creating and executing complex document processing and data transformation pipelines powered by large language models, featuring an interactive UI playground and a Python package for production use.
hitsz-ids/synthetic-data-generator
Synthetic Data Generator (SDG) is a comprehensive framework for generating high-quality, privacy-compliant synthetic structured tabular data using advanced statistical and LLM-based models, optimized for big data applications.
Mintplex-Labs/vector-admin
VectorAdmin is a universal and user-friendly tool suite for managing multiple vector databases, providing full control over vector data with multi-user support, embedding management, and cloud deployment capabilities.