DataFlow

DataFlow is an agent-based workflow orchestrator and data pipeline designed to synthesize, clean, and augment large-scale datasets for training large language models. It functions as a synthetic data generator and text curation tool, utilizing an intelligent assistant to assemble modular processing operators into functional pipelines based on user requirements.

The project distinguishes itself through a low-code approach, providing a web-based visual interface for designing and monitoring multi-stage execution flows. It features an operator-based registry system that allows for the integration of third-party components and a guided command-line process to bootstrap distributable operator libraries.

The platform covers a broad range of data engineering capabilities, including unstructured knowledge extraction, text deduplication via MinHash, and noise filtering. It specifically supports the generation of complex reasoning chains, multi-hop question-answer pairs, and synthetic SQL datasets with integrated validity filtering and difficulty evaluation.

The system is implemented in Python.

Features

Training Datasets - Generates and cleans large-scale, high-quality datasets specifically for training large language models.
Training Data Generation - Generates and refines high-quality question-answer pairs and reasoning chains to create diverse LLM training datasets.
Agent-Based Pipeline Assembly - Provides an intelligent assistant to assemble modular operators into functional data processing pipelines.
Agentic Workflow Automation - Uses an intelligent agent to automate the assembly of existing operators into new processing pipelines.
Agentic Workflow Orchestrators - Features an intelligent assistant that assembles modular processing operators into functional data pipelines.
Text Dataset Curators - Filters, formats, and deduplicates unstructured text data using MinHash and noise filtering for LLM training.
Text-to-SQL Dataset Synthesis - Synthesizes complex question-answer pairs and reasoning chains from SQL samples and schemas for model training.
Training Data Pipelines - Orchestrates the synthesis, cleaning, and augmentation of large-scale datasets used to train large language models.
Intra-Dataset Deduplication - Identifies and removes redundant data entries using the MinHash algorithm to ensure dataset diversity.
Data Pipeline Orchestration - Provides a system for defining, scheduling, and monitoring complex sequences of modular data processing tasks.
Multi-Stage Pipeline Processing - Composes modular operators into multi-stage workflows to systematically generate and clean datasets.
Browser-Based and Visual Development Interfaces - Offers a web-based visual interface for designing, mapping, and monitoring low-code data pipelines.
Multi-Stage Pipeline Orchestrators - Orchestrates multi-stage pipelines to systematically generate, clean, and augment large-scale datasets.
Modular Extension Registries - Maintains a central registry of modular data processing operators to allow extensibility and reuse across pipelines.
Visual Pipeline Builders - Provides a graphical interface for the low-code construction and execution of data pipelines.
Answer Filtering - Filters generated answers by format and length and verifies accuracy against ground truth.
Document Knowledge Extraction - Converts unstructured sources such as PDFs, Word documents, and tables into organized QA pairs or entries.
Execution Consistency Voting - Implements execution-consistency voting to select the most accurate reasoning logic across multiple generated chains.
Text Chunks - Divides long markdown text into smaller segments based on token, character, sentence, or semantic dimensions.
Prompt Templates - Formats database schemas and questions into structured prompt templates for model training.
Reasoning Chain Generation - Produces detailed long-chain reasoning processes or pseudo-answers via model voting for mathematical questions.
Multi-Hop Question Generators - Synthesizes human-like questions and supporting evidence from SQL statements to create complex QA pairs.
Training Pair Generators - Creates questions and multiple variations of answers from raw text to provide structured training data.
Reasoning Chains - Produces detailed chain-of-thought reasoning for translating natural language questions into SQL.
Reasoning Problem Synthesis - Generates new mathematical problem variants based on existing data to increase dataset scale and diversity.
Synthetic Data Curation Tools - Includes utilities for scoring and filtering synthesized text based on style, educational value, and factual accuracy.
Synthetic Reasoning Data Generators - Produces high-quality reasoning chains and QA pairs using model-based synthesis and consistency voting.
Data Analysis Agents - Employs intelligent agents to analyze data, write custom operators, and organize processing pipelines.
Document to Markdown Converters - Extracts textual knowledge from various file formats or URLs and transforms them into a unified markdown format.
Custom Pipeline Orchestration - Enables the combination of core and custom operators to build distributable data processing workflows.
Rule-Based Data Filtering - Applies deterministic constraints on length, language, and symbols to remove noise from raw datasets.
SQL Query Generation - Provides automated creation of SQL queries based on database schemas to increase dataset diversity.
Synthetic Dataset Generators - Creates and validates SQL queries and corresponding natural language questions for Text-to-SQL training.
Text Cleaning Pipelines - Removes noise such as emojis, extra spaces, HTML tags, and URLs to refine raw text quality.
Text Quality Filtering - Drops documents based on language, word count, punctuation, symbol ratios, or forbidden words.
MinHash-LSH Indexing - Utilizes MinHash-LSH indexing to identify and remove near-duplicate text entries for training set diversity.
Pipeline Extenders - Allows the registration of external operator packages into a central registry to extend pipeline capabilities.
Response Quality Scoring - Evaluates the quality of text samples using model-based scoring to retain only high-value content.
SQL Logic Validators - Filters out incorrect SQL statements by verifying their functional logic and consistency against a database schema.
Data Pipeline Flow Designers - Provides a web-based visual interface for designing and monitoring the execution of multi-stage data processing flows.
Data Processing - Data preparation pipelines using language model operators.
Data Processing Tools - Data preparation pipelines using model-based operators.
Workflow Orchestration - Platform for automating data preparation and AI pipeline workflows.

camel-ai/camel

17,253View on GitHub

This project is a comprehensive framework for building and managing autonomous agent systems. It provides a unified architecture for orchestrating multi-agent societies, where specialized agents collaborate through roleplay to decompose and solve complex tasks. The system integrates language models with external environments, enabling agents to perform real-world actions through a standardized tool-calling abstraction layer. The framework distinguishes itself through its focus on iterative reasoning and data reliability. It employs automated feedback loops to refine agent outputs and self-eva

apache/nifi

5,976View on GitHub

Apache NiFi is a flow-based programming platform that enables the visual design, monitoring, and management of data pipelines. At its core, it provides a web-based visual dataflow designer where users build directed graphs of processors to route, transform, and mediate data movement between any source and destination without writing custom code. The system records fine-grained data provenance for every data item from ingestion to delivery, supporting audit, debugging, and replay of data lineage. The platform distinguishes itself through a zero-master cluster architecture that distributes proc

vibrantlabsai/ragas

12,659View on GitHub

Ragas is an evaluation framework designed to measure the performance of retrieval-augmented generation pipelines and autonomous agent workflows. It provides a comprehensive suite of tools for benchmarking system outputs, utilizing language models as automated judges to score performance against defined rubrics and reference data. By standardizing inputs, retrieved contexts, and generated responses into a unified schema, the project enables consistent analysis across complex AI applications. The framework distinguishes itself through its ability to generate synthetic test datasets from existin

steveyegge/beads

16,757View on GitHub

Beads is a versioned, dependency-aware graph database designed for distributed issue tracking and project management. It functions as an agentic workflow orchestrator, providing a structured environment where tasks, dependencies, and project metadata are linked through relational hierarchies. By maintaining a persistent, version-controlled record of project state, the system enables teams to manage complex work items across multiple repositories and environments. The platform distinguishes itself through its deep integration with automated coding agents, acting as a Model Context Protocol ser

OpenDCAIDataFlow

Features