meta-llama/synthetic-data-kit

Features

• Synthetic Data Generators - Creates and curates high-quality training datasets from raw documents using language model inference.
• Modular Pipeline Orchestrators - Orchestrates modular processing stages to transform raw documents into structured training datasets.
• Document Processing Pipelines - Provides an end-to-end pipeline for extracting and structuring text from raw documents.
• Machine Learning Pipelines - Provides a modular orchestration engine that manages the end-to-end lifecycle of transforming raw documents into structured machine learning training datasets.
• LLM-As-A-Judge Scoring - Uses language models as judges to evaluate and filter generated content for quality and accuracy.
• Synthetic Dataset Generators - Automates the generation of diverse synthetic training datasets, including summaries and question-answer pairs, through language model inference on source documents.
• Text Dataset Curators - Curates machine learning datasets through automated quality filtering and validation.
• Multimodal Feature Extractors - Provides a unified layer for extracting text and image content from mixed-media documents.
• Tool-Use Training - Produces specialized training data that teaches language models how to effectively interact with external software tools and APIs.
• LLM Fine-Tuning Toolsets - Prepares high-quality, structured training datasets specifically for language model fine-tuning.
• Dataset Curation Tools - Provides utilities for evaluating and filtering generated content to ensure training suitability.
• Multimodal Document Processing - Extracts text and image content from mixed-media documents to support synthetic data generation.
• Prompt Templates - Injects source content and instructions into configurable templates to drive dynamic generation behavior.
• Semantic Chunking - Splits large source documents into overlapping segments to maintain semantic coherence during synthetic data generation.
• Synthetic Reasoning Data Generators - Generates structured chains of thought and reasoning traces to enhance the logical problem-solving capabilities of language models.
• Training Data Exporters - Exports processed datasets into standardized formats for downstream model training.
• Document Parsing Pipelines - Parses diverse file formats into structured text for downstream processing.
• Multimodal Quality Filters - Filters generated content using automated quality checks to ensure high-quality training samples.

سجل النجوم