hitsz-ids/synthetic-data-generator
Synthetic Data Generator
该项目是一个用于生成合成表格数据的框架，该数据保留了原始源数据集的统计属性和关系完整性。它作为一个元数据驱动的引擎，利用语言模型来合成信息，即使在原始训练样本受限的情况下也是如此。该系统旨在在复杂的、多表结构中保持逻辑一致性，同时确保生成的输出符合定义的模式要求。
该平台通过专注于隐私保护合成而脱颖而出，集成了通过差分隐私和匿名化技术量化并减轻重新识别风险的工具。它支持模块化扩展，允许集成自定义生成模型和数据连接器。此外，该框架包括自动化验证例程，将合成输出的分布和相关模式与源数据进行比较，以验证统计保真度。
除了核心生成外，该系统还通过从学习到的模式中派生新列，提供了数据增强和特征工程的功能。它结合了操作监督工具，以在海量任务期间监控资源利用率和处理效率。该库旨在通过内存高效的流处理和迭代批处理来处理大规模数据集，以确保稳定性。
Features
- Tabular - Creates privacy-preserving synthetic tabular records that maintain the statistical properties of original source data.
- Data Synthesis - Leverages language models and metadata to synthesize information when original training samples are restricted.
- Synthetic Data Generators - Creates privacy-preserving structured datasets that maintain the statistical properties of original source data.
- Synthetic Data Generators - Generates synthetic datasets using metadata and language model knowledge without requiring original training samples.
- Relational Synthesis - Generates consistent synthetic datasets across multiple related tables while preserving structural integrity and logical relationships.
- Statistical Fidelity Evaluators - Validates synthetic data quality by comparing distribution and correlation patterns against original source datasets.
- Privacy and Data Protection - Protects sensitive information using differential privacy and masking techniques to ensure compliance.
- Differential Privacy Aggregators - Provides a toolkit for quantifying and mitigating re-identification risks in synthetic datasets.
- Differential Privacy Noise Injection - Injects mathematical noise into synthetic data generation to ensure individual records cannot be re-identified.
- Schema-Driven Generators - Constructs synthetic datasets by interpreting structured schema definitions rather than relying on raw input samples.
- Relational Data Generators - Generates consistent synthetic data across multiple related tables while preserving structural integrity.
- Synthetic Data Generation - Enriches datasets by generating new columns based on learned patterns to improve data depth.
- Data Enrichment - Enriches datasets by deriving new columns from learned patterns and external knowledge.
- Security & Privacy - Identifies personal identifiers and applies privacy techniques to secure generated information.
- Memory-Efficient Data Streaming - Processes large-scale datasets in memory-efficient chunks to maintain system stability during high-volume generation.
- Graph-Relational Models - Maintains logical consistency across multi-table structures by enforcing structural dependencies during data generation.
- Relational Data Modeling - Models complex multi-table relationships to ensure logical consistency during synthetic data generation.
- Generation Framework Extensions - Allows integration of custom generation models, data processing routines, and external connectors through a modular plugin architecture.
