DataFlow

**Generate, Clean, and Prepare LLM Data, All-in-One** [](https://github.com/OpenDCAI/DataFlow) [](https://github.com/OpenDCAI/DataFlow/issues) [](https://github.com/OpenDCAI/DataFlow/issues?q=is%3Aissue%20state%3Aclosed) [](https://github.com/OpenDCAI/DataFlow/pulls) [](https://github.com/OpenDCAI/DataFlow/pulls?q=is%3Apr+is%3Aclosed) [](https://github.com/OpenDCAI/DataFlow/graphs/contributors) [](https://github.com/OpenDCAI/DataFlow) [](https://pypi.org/project/open-dataflow/) [](https://pypi.org/project/open-dataflow/) [](https://pypistats.org/packages/open-dataflow) [](https://pepy.tech/project/open-dataflow) [](https://deepwiki.com/OpenDCAI/DataFlow) [](https://colab.research.google.com/drive/1haosl2QS4N4HM7u7HvSsz_MnLabxexXl?usp=sharing) [](https://hub.docker.com/r/molyheci/dataflow) [](https://OpenDCAI.github.io/DataFlow-Doc/) [](https://arxiv.org/abs/2512.16676) [](https://2027.sigmod.org/) [](https://discord.gg/e4mKEaFptu) [](https://github.com/user-attachments/assets/3c2e5d4d-d1ea-4d8c-9146-ff14e657e857) Visual, low-code pipelines with flexible orchestration across domains and use cases.💪 Turn raw data into high-quality LLM training datasets.🔧 🎉 Get smarter LLMs cheaply — give us a star ⭐ on GitHub for the latest update. **Beginner-friendly learning resources (continuously updated)**: [[🎬 Video Tutorials]](https://space.bilibili.com/3546929239689711?spm_id_from=333.337.0.0) [[📚 Written Tutorials]](https://wcny4qa9krto.feishu.cn/wiki/I9tbw2qnBi0lEakmmAGclTysnFd) [简体中文](https://github.com/OpenDCAI/DataFlow/blob/HEAD/./README-zh.md) | English

📰 0. News

• [2026-09-12] 🎉 DataFlow has been accepted by ACM SIGMOD 2027! Our paper, DataFlow: An LLM-Driven Framework for Unified Data Preparation and Workflow Automation in the Era of Data-Centric AI, has been accepted by ACM SIGMOD 2027.
• [2026-07-18] DataFlow-Harness Released! 🔥DataFlow-Harness lets coding agents build DataFlow pipelines for you. Download and installation entry points are available in DataFlow-WebUI, where you can use DataFlow through either the visual interface or MCP.
• [2026-05-22] DataFlow-Skills is now available! A collection of skills and tutorials for working with DataFlow. 👉 DataFlow-Skills

• [2026-02-02] 🖥️ DataFlow WebUI is now available! Launch the visual pipeline builder with a single command: dataflow webui. Build and run DataFlow pipelines through an intuitive web interface. 👉 WebUI Docs
  

  

• [2026-01-20] 🌟 Awesome Works Using DataFlow is now live! A new section showcasing open-source projects and research built on DataFlow. Contributions are welcome! 👉 Awesome Works

• [2025-12-19] 🎉 Our DataFlow technical report is now available! Read and cite our work on arXiv: https://arxiv.org/abs/2512.16676

• [2025-11-20] 🤖 Introducing New Data Agents for DataFlow! Try them out and follow the tutorial on Bilibili: https://space.bilibili.com/3546929239689711/lists/6761342?type=season

• [2025-06-28] 🎉 DataFlow is officially released! Our data-centric AI system is now public. Stay tuned for future updates.

🔍 1. What is DataFlow？

DataFlow is a data preparation and training system designed to generate, refine, evaluate, and filter high-quality data for AI from noisy sources (PDF, plain-text, low-quality QA), thereby improving the performance of large language models (LLMs) in specific domains through targeted training (Pre-training, Supervised Fine-tuning, RL training) or RAG system, in domains such as healthcare, finance, legal, and academic research.

Through an operator-based design, DataFlow turns the entire data cleaning workflow into a reproducible, reusable, and shareable pipeline, providing core infrastructure for the Data-Centric AI community. Additionally, we develop an intelligent DataFlow-agent capable of dynamically assembling new pipelines by recombining existing or creating new operators on demand.

🔍 2. Key Features

✅2.1 Ready-to-Use Data Synthesis and Cleaning Pipelines

• High-Quality Training Data Generation
• Text, Math, and Code data generation (see DataFlow-Instruct-10K for results)
• Data generation via tools like AgenticRAG and Text2SQL
• Structured Data Extraction
• Large-scale PDF → QA conversion
• Large-scale book PDF → Visual-QA conversion
• Scientific Data Workflow Management
• Text2SQL workflow management (Accepted by ICDE 2026)
• Math data workflows (Accepted by KDD 2026)

⚙️2.2 Flexible Custom Pipeline Orchestration

• 10+ core operators define interaction patterns and design principles
• 100+ pipeline-specific operators available for reuse or reference
• Full support for creating custom operators — plug-and-play, easily packaged and distributed via GitHub or PyPI

🧠2.3 Reproducible, Reusable, and Shareable Data-Centric AI System

• Data governance algorithms are encapsulated as operator pipelines, enabling reproducibility and fair comparison of different data governance strategies (❤️research-friendly)
• Easily reuse swap underlying large models to analyze the relationship between model performance and data quality quickly
• Built on Python and Git ecosystems for easy distribution, management, and traceability of high-quality, user-defined data governance operators and pipelines (❤️enterprise-friendly)

🛠️ 3. DataFlow Suite

The DataFlow Suite provides the essential infrastructure to automate and scale LLM data preparation with DataFlow main repository. It comprises four tightly integrated layers: - DataFlow-Skills – A collection of skills and tutorials for working with DataFlow, covering operator development, pipeline construction, and best practices for data-centric AI.

• DataFlow-WebUI – A visual and AI Agent assisted workspace for building, managing, and running DataFlow pipelines. It combines drag-and-drop pipeline editing with natural-language interaction, enabling users to construct and execute data workflows more efficiently.

• DataFlow-Ecosystem – A modular distribution layer that standardizes operator registration. It enables domain-specific modules (e.g., DataFlow-MM, DataFlow-AI4S) to contribute extensible libraries under a unified abstraction.

• RayOrch – A high-performance orchestration layer built on Ray, providing distributed compute scheduling and resource management for massive-scale data tasks.

Together, these components form a unified, extensible environment that transforms raw data into model-ready intelligence.

✅ 4. Why use DataFlow?

Data generation and cleaning ar