LLM Synthetic Data Generation
For a tool for generating synthetic datasets, the strongest matches are conardli/easy-dataset (Easy-dataset is an end-to-end platform for managing ML datasets), tatsu-lab/stanford_alpaca (Stanford Alpaca is an end-to-end framework that uses LLMs) and meta-llama/synthetic-data-kit (This tool from Meta is purpose-built for generating high-quality). camel-ai/owl and opendcai/dataflow round out the shortlist. Each is ranked by relevance to your query, popularity and recent activity.
Frameworks and tools for creating high-quality synthetic datasets to train and fine-tune large language models.
LLM Synthetic Data Generation
Find the best repos with AI.We'll search the best matching repositories with AI.
- conardli/easy-dataset
ConardLi/easy-dataset
13,394View on GitHubEasy-dataset is a comprehensive platform designed for the end-to-end management of machine learning datasets, specifically tailored for language and vision model fine-tuning. It functions as a centralized environment for the entire data lifecycle, encompassing the automated generation of synthetic training data, the structural organization of document collections, and the systematic annotation of individual data points. The platform distinguishes itself through its integrated evaluation and orchestration capabilities. It provides a dedicated suite for benchmarking models, featuring blind side
Easy-dataset is an end-to-end platform for managing ML datasets that explicitly covers automated synthetic data generation with LLM integration, quality filtering, and batch processing, directly matching the search for a synthetic data generation framework.
JavaScriptSynthetic Data PipelinesSynthetic Data Generators - tatsu-lab/stanford_alpaca
tatsu-lab/stanford_alpaca
30,266View on GitHubThis project provides an end-to-end framework for adapting large language models to follow user instructions through supervised fine-tuning. It functions as a comprehensive training pipeline that enables the creation of specialized assistant models by minimizing the difference between predicted outputs and target responses within structured instruction datasets. The framework distinguishes itself by integrating synthetic data generation with memory-efficient training techniques. It utilizes powerful language models to iteratively expand small sets of human-written seeds into diverse, high-qua
Stanford Alpaca is an end-to-end framework that uses LLMs to iteratively expand small seed sets into diverse synthetic instruction-following datasets, making it a clear fit for generating synthetic training data with LLMs.
PythonSynthetic Data Generators - meta-llama/synthetic-data-kit
meta-llama/synthetic-data-kit
1,602View on GitHubThe synthetic data kit is an integrated framework designed to generate, curate, and format training datasets for language models. It provides an end-to-end pipeline that transforms raw source documents into structured data suitable for fine-tuning, reasoning, and tool-use model training. The framework distinguishes itself through a modular orchestration engine that manages the entire lifecycle of data preparation. It supports multimodal input by extracting both text and image content from various file formats, while employing context-aware chunking to maintain semantic coherence. The generati
This tool from Meta is purpose-built for generating high-quality synthetic datasets using LLMs, fitting the requirement for an open-source framework that integrates LLM APIs and supports prompt-driven generation.
PythonSynthetic Data Generators - camel-ai/owl
camel-ai/owl
19,864View on GitHubOwl is a framework for agentic workflow automation and multi-agent orchestration. It functions as a system for coordinating autonomous large language model agents to decompose and execute complex tasks through shared communication and collaborative planning. The project distinguishes itself through a multi-modal toolset for processing images, audio, and video, alongside a synthetic data generator that produces domain-specific datasets using self-instruct and verifier loops. It further incorporates a retrieval-augmented generation pipeline framework that integrates long-term memory and real-ti
Owl is an agent orchestration framework that includes a synthetic data generator using LLM self-instruct and verifier loops, so it can produce domain-specific datasets — fitting the core need for synthetic data generation, though its primary focus is broader.
PythonSynthetic Data Generators - opendcai/dataflow
OpenDCAI/DataFlow
2,926View on GitHubDataFlow is an agent-based workflow orchestrator and data pipeline designed to synthesize, clean, and augment large-scale datasets for training large language models. It functions as a synthetic data generator and text curation tool, utilizing an intelligent assistant to assemble modular processing operators into functional pipelines based on user requirements. The project distinguishes itself through a low-code approach, providing a web-based visual interface for designing and monitoring multi-stage execution flows. It features an operator-based registry system that allows for the integratio
DataFlow is an agent-based workflow orchestrator for synthesizing, cleaning, and augmenting large-scale LLM training datasets, with LLM backend integration (vLLM, SGLang) and a low-code visual pipeline designer, directly addressing your need for a synthetic data generation framework.
PythonText Quality Filtering - datajuicer/data-juicer
datajuicer/data-juicer
6,574View on GitHubData-Juicer is an open-source framework for cleaning, filtering, deduplicating, and transforming multimodal datasets to prepare them for training large language and vision models. It functions as a distributed data pipeline engine that runs processing jobs across Ray clusters, handling billions of samples with automatic operator fusion and adaptive parallelism. The framework provides a library of operators that leverage large language models for semantic extraction, filtering, and data synthesis within processing pipelines. The project distinguishes itself through a YAML-based data recipe sys
Data-Juicer is a pipeline framework that includes LLM-powered operators for data synthesis and filtering, making it a valid tool for generating synthetic training data, though its primary focus is on broader multimodal data processing rather than dedicated generation.
PythonText Quality Filtering - argilla-io/distilabel
argilla-io/distilabel
3,277View on GitHubDistilabel is a framework for synthetic data and AI feedback for engineers who need fast, reliable and scalable pipelines based on verified research papers.
Distilabel is a framework for building scalable synthetic data and AI feedback pipelines using LLMs, directly matching the search for an open-source tool that generates training data with LLM integration and batch processing.
PythonData Generation FrameworksData ProcessingData Processing Tools - bespokelabsai/curator
bespokelabsai/curator
1,637View on GitHubCurator is a Python framework for generating synthetic datasets using LLMs, fitting the search for a synthetic data generation tool with LLM integration.
PythonData Generation FrameworksData Processing - huggingface/open-r1
huggingface/open-r1
26,326View on GitHubOpen-r1 is a framework designed for the large-scale training, distillation, and optimization of language models focused on complex reasoning and programming tasks. It provides a comprehensive suite of tools for managing distributed training jobs across multi-node clusters, enabling the development of high-performance models through reinforcement learning and supervised fine-tuning. The project distinguishes itself by integrating secure, containerized code execution environments directly into the training and evaluation lifecycle. By allowing models to run and verify code snippets against test
Open-r1 is primarily a framework for training and optimizing reasoning models, not a dedicated tool for generating synthetic data via LLM APIs, even though it includes some data generation and filtering components.
PythonSynthetic Data PipelinesSynthetic Data Generators - togethercomputer/redpajama-data
togethercomputer/RedPajama-Data
4,947View on GitHubRedPajama-Data is a toolset for preprocessing large-scale text datasets used to train large language models. It provides a preprocessing pipeline focused on cleaning, deduplicating, and scoring massive collections of text to ensure data quality and diversity. The project utilizes a document quality scoring framework that employs machine learning and statistical heuristics to evaluate if documents are suitable for training. It includes a dataset filtering pipeline that uses classifiers and blocklists to remove undesirable words or URLs. The system features a text deduplication toolset that el
RedPajama-Data preprocesses existing text datasets for LLM training rather than generating new synthetic data using LLMs, so it fits the upstream data-quality domain but not the generative pipeline you're searching for.
PythonText Quality Filtering - alibaba-nlp/webagent
Alibaba-NLP/WebAgent
19,549View on GitHubWebAgent is an autonomous web navigation agent and research system designed to browse the internet and synthesize information to answer complex queries. It functions as a reasoning orchestrator that navigates the web iteratively to perform deep research and extract structured data. The project includes a reinforcement learning training pipeline that generates synthetic interaction datasets for model pre-training and fine-tuning. It employs token-level policy gradients to stabilize training in non-stationary environments and uses a dual-mode inference scaling mechanism to balance execution bet
WebAgent is primarily an autonomous web navigation agent that uses synthetic interaction datasets for model training, rather than a dedicated tool for generating diverse synthetic training data with LLMs across multiple formats.
PythonSynthetic Data Generators - eriklindernoren/keras-gan
eriklindernoren/Keras-GAN
9,206View on GitHubKeras-GAN is a collection of generative adversarial network implementations built with Keras for synthetic data generation and image manipulation. It provides frameworks for image-to-image translation, image inpainting, and neural image super-resolution. The library includes tools for learning disentangled latent space representations to control specific attributes of synthetic outputs. It also features capabilities for image domain translation using paired or unpaired data and the ability to fill corrupted or missing image parts by analyzing surrounding visual context. The project covers ge
Keras-GAN provides GAN-based image generation models, not LLM-driven synthetic text data generation with prompt pipelines or API integration, so it falls outside the specific category the visitor needs.
PythonSynthetic Data Generators
