meta-llama/synthetic-data-kit
Synthetic Data Kit
The synthetic data kit is an integrated framework designed to generate, curate, and format training datasets for language models. It provides an end-to-end pipeline that transforms raw source documents into structured data suitable for fine-tuning, reasoning, and tool-use model training.
The framework distinguishes itself through a modular orchestration engine that manages the entire lifecycle of data preparation. It supports multimodal input by extracting both text and image content from various file formats, while employing context-aware chunking to maintain semantic coherence. The generation process is driven by template-based prompt injection, and the resulting output is validated through an automated evaluation system that uses language models as judges to ensure quality and accuracy.
The project covers a broad range of data processing capabilities, including document parsing, automated quality filtering, and schema-agnostic serialization. It supports the creation of diverse training examples, such as reasoning traces and tool-use demonstrations, and exports the final datasets into standardized formats for compatibility with machine learning training frameworks.
Users manage the generation workflow and pipeline stages through centralized configuration files and command-line arguments.
Features
- Synthetic Data Generators - Creates and curates high-quality training datasets from raw documents using language model inference.
- Modular Pipeline Orchestrators - Orchestrates modular processing stages to transform raw documents into structured training datasets.
- Document Processing Pipelines - Provides an end-to-end pipeline for extracting and structuring text from raw documents.
- Machine Learning Pipelines - Provides a modular orchestration engine that manages the end-to-end lifecycle of transforming raw documents into structured machine learning training datasets.
- LLM-As-A-Judge Scoring - Uses language models as judges to evaluate and filter generated content for quality and accuracy.
- Synthetic Dataset Generators - Automates the generation of diverse synthetic training datasets, including summaries and question-answer pairs, through language model inference on source documents.
- Text Dataset Curators - Curates machine learning datasets through automated quality filtering and validation.
- Multimodal Feature Extractors - Provides a unified layer for extracting text and image content from mixed-media documents.
- Tool-Use Training - Produces specialized training data that teaches language models how to effectively interact with external software tools and APIs.
- LLM Fine-Tuning Toolsets - Prepares high-quality, structured training datasets specifically for language model fine-tuning.
- Dataset Curation Tools - Provides utilities for evaluating and filtering generated content to ensure training suitability.
- Multimodal Document Processing - Extracts text and image content from mixed-media documents to support synthetic data generation.
- Prompt Templates - Injects source content and instructions into configurable templates to drive dynamic generation behavior.
- Semantic Chunking - Splits large source documents into overlapping segments to maintain semantic coherence during synthetic data generation.
- Synthetic Reasoning Data Generators - Generates structured chains of thought and reasoning traces to enhance the logical problem-solving capabilities of language models.
- Training Data Exporters - Exports processed datasets into standardized formats for downstream model training.
- Document Parsing Pipelines - Parses diverse file formats into structured text for downstream processing.
- Multimodal Quality Filters - Filters generated content using automated quality checks to ensure high-quality training samples.
