OpenDCAI/DataFlow

Features

• Training Datasets - Generates and cleans large-scale, high-quality datasets specifically for training large language models.
• Training Data Generation - Generates and refines high-quality question-answer pairs and reasoning chains to create diverse LLM training datasets.
• Agent-Based Pipeline Assembly - Provides an intelligent assistant to assemble modular operators into functional data processing pipelines.
• Agentic Workflow Automation - Uses an intelligent agent to automate the assembly of existing operators into new processing pipelines.
• Agentic Workflow Orchestrators - Features an intelligent assistant that assembles modular processing operators into functional data pipelines.
• Text Dataset Curators - Filters, formats, and deduplicates unstructured text data using MinHash and noise filtering for LLM training.
• Text-to-SQL Dataset Synthesis - Synthesizes complex question-answer pairs and reasoning chains from SQL samples and schemas for model training.
• Training Data Pipelines - Orchestrates the synthesis, cleaning, and augmentation of large-scale datasets used to train large language models.
• Intra-Dataset Deduplication - Identifies and removes redundant data entries using the MinHash algorithm to ensure dataset diversity.
• Data Pipeline Orchestration - Provides a system for defining, scheduling, and monitoring complex sequences of modular data processing tasks.
• Multi-Stage Pipeline Processing - Composes modular operators into multi-stage workflows to systematically generate and clean datasets.
• Browser-Based and Visual Development Interfaces - Offers a web-based visual interface for designing, mapping, and monitoring low-code data pipelines.
• Multi-Stage Pipeline Orchestrators - Orchestrates multi-stage pipelines to systematically generate, clean, and augment large-scale datasets.
• Modular Extension Registries - Maintains a central registry of modular data processing operators to allow extensibility and reuse across pipelines.
• Visual Pipeline Builders - Provides a graphical interface for the low-code construction and execution of data pipelines.
• Answer Filtering - Filters generated answers by format and length and verifies accuracy against ground truth.
• Document Knowledge Extraction - Converts unstructured sources such as PDFs, Word documents, and tables into organized QA pairs or entries.
• Execution Consistency Voting - Implements execution-consistency voting to select the most accurate reasoning logic across multiple generated chains.
• Text Chunks - Divides long markdown text into smaller segments based on token, character, sentence, or semantic dimensions.
• Prompt Templates - Formats database schemas and questions into structured prompt templates for model training.
• Reasoning Chain Generation - Produces detailed long-chain reasoning processes or pseudo-answers via model voting for mathematical questions.
• Multi-Hop Question Generators - Synthesizes human-like questions and supporting evidence from SQL statements to create complex QA pairs.
• Training Pair Generators - Creates questions and multiple variations of answers from raw text to provide structured training data.
• Reasoning Chains - Produces detailed chain-of-thought reasoning for translating natural language questions into SQL.
• Reasoning Problem Synthesis - Generates new mathematical problem variants based on existing data to increase dataset scale and diversity.
• Synthetic Data Curation Tools - Includes utilities for scoring and filtering synthesized text based on style, educational value, and factual accuracy.
• Synthetic Reasoning Data Generators - Produces high-quality reasoning chains and QA pairs using model-based synthesis and consistency voting.
• Data Analysis Agents - Employs intelligent agents to analyze data, write custom operators, and organize processing pipelines.
• Document to Markdown Converters - Extracts textual knowledge from various file formats or URLs and transforms them into a unified markdown format.
• Custom Pipeline Orchestration - Enables the combination of core and custom operators to build distributable data processing workflows.
• Rule-Based Data Filtering - Applies deterministic constraints on length, language, and symbols to remove noise from raw datasets.
• SQL Query Generation - Provides automated creation of SQL queries based on database schemas to increase dataset diversity.
• Synthetic Dataset Generators - Creates and validates SQL queries and corresponding natural language questions for Text-to-SQL training.
• Text Cleaning Pipelines - Removes noise such as emojis, extra spaces, HTML tags, and URLs to refine raw text quality.
• Text Quality Filtering - Drops documents based on language, word count, punctuation, symbol ratios, or forbidden words.
• MinHash-LSH Indexing - Utilizes MinHash-LSH indexing to identify and remove near-duplicate text entries for training set diversity.
• Pipeline Extenders - Allows the registration of external operator packages into a central registry to extend pipeline capabilities.
• Response Quality Scoring - Evaluates the quality of text samples using model-based scoring to retain only high-value content.
• SQL Logic Validators - Filters out incorrect SQL statements by verifying their functional logic and consistency against a database schema.
• Data Pipeline Flow Designers - Provides a web-based visual interface for designing and monitoring the execution of multi-stage data processing flows.
• Data Processing - Data preparation pipelines using language model operators.
• Data Processing Tools - Data preparation pipelines using model-based operators.
• Workflow Orchestration - Platform for automating data preparation and AI pipeline workflows.

Star history