nucleuscloud/neosyncArchived
Neosync
NeoSync is a database synchronization tool and data pipeline orchestrator designed to move and transform datasets across different environments. It functions as a PII data security platform and a synthetic data generator, allowing for the synchronization of production data while ensuring privacy compliance.
The system utilizes an event-sourced coordinator to manage asynchronous data movements, providing automated retry and failure handling. It differentiates itself by combining rule-based PII anonymization and detection with schema-based synthetic data generation to create artificial datasets that mimic production properties without exposing private information.
The project covers broad capability areas including database subsetting to reduce data volume for testing, template-driven field transformations to reshape information, and the orchestration of data pipelines to maintain relational integrity during synchronization.
Features
- Database Environment Synchronization - Synchronizes and transforms data slices between production and development environments with built-in failure management.
- Synthetic Data Generators - Creates artificial datasets that mimic production properties for unit testing, seeding, and demonstrations.
- Relational Database Subsetting - Extracts specific data slices from production databases while maintaining referential integrity for local development.
- Data Pipeline Orchestration - Coordinates asynchronous data movements and processing tasks with automated failure recovery.
- Database Synchronization Tools - Syncs production data across environments while applying subsetting, anonymization, and transformation rules.
- Production Data Subsetting - Extracts representative record slices from production databases to reduce data volume for local testing.
- Data Anonymization - Removes personally identifiable information from production datasets to enable safe local development and testing.
- PII Detection and Screening - Identifies and masks personally identifiable information using pattern matching and replacement logic to ensure privacy compliance.
- PII Security Platforms - Offers a comprehensive platform to monitor, detect, and anonymize PII within production datasets for privacy compliance.
- Pipeline Orchestration - Implements an event-sourced coordinator to ensure reliable data pipeline execution with automated retry and failure handling.
- Synthetic Dataset Generation - Produces artificial datasets by analyzing database schemas to maintain relational integrity without using real production data.
- Field Transformations - Modifies specific data columns during synchronization using predefined scripts or models to reshape information.
- Background Data Synchronization - Provides background workers to move datasets between environments without blocking the main application execution.
