# nucleuscloud/neosync

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/nucleuscloud-neosync).**

4,149 stars · 231 forks · Go · NOASSERTION · archived

## Links

- GitHub: https://github.com/nucleuscloud/neosync
- Homepage: https://www.neosync.dev
- awesome-repositories: https://awesome-repositories.com/repository/nucleuscloud-neosync.md

## Topics

`benthos` `docker` `etl` `faker` `fine-tuning` `golang` `kubernetes` `mysql` `nextjs` `open-source` `orchestration` `postgresql` `reactjs` `self-hosted` `synthetic-data` `synthetic-data-generation` `test-data-generator` `testing` `typescript`

## Description

NeoSync is a database synchronization tool and data pipeline orchestrator designed to move and transform datasets across different environments. It functions as a PII data security platform and a synthetic data generator, allowing for the synchronization of production data while ensuring privacy compliance.

The system utilizes an event-sourced coordinator to manage asynchronous data movements, providing automated retry and failure handling. It differentiates itself by combining rule-based PII anonymization and detection with schema-based synthetic data generation to create artificial datasets that mimic production properties without exposing private information.

The project covers broad capability areas including database subsetting to reduce data volume for testing, template-driven field transformations to reshape information, and the orchestration of data pipelines to maintain relational integrity during synchronization.

## Tags

### Data & Databases

- [Database Environment Synchronization](https://awesome-repositories.com/f/data-databases/database-environment-synchronization.md) — Synchronizes and transforms data slices between production and development environments with built-in failure management.
- [Data Pipeline Orchestration](https://awesome-repositories.com/f/data-databases/data-pipeline-orchestration.md) — Coordinates asynchronous data movements and processing tasks with automated failure recovery. ([source](https://cdn.jsdelivr.net/gh/nucleuscloud/neosync@main/README.md))
- [Database Synchronization Tools](https://awesome-repositories.com/f/data-databases/database-synchronization-tools.md) — Syncs production data across environments while applying subsetting, anonymization, and transformation rules.
- [Production Data Subsetting](https://awesome-repositories.com/f/data-databases/production-data-subsetting.md) — Extracts representative record slices from production databases to reduce data volume for local testing. ([source](https://cdn.jsdelivr.net/gh/nucleuscloud/neosync@main/README.md))
- [Field Transformations](https://awesome-repositories.com/f/data-databases/field-transformations.md) — Modifies specific data columns during synchronization using predefined scripts or models to reshape information.

### Artificial Intelligence & ML

- [Synthetic Data Generators](https://awesome-repositories.com/f/artificial-intelligence-ml/dataset-generation/synthetic-data-generators.md) — Creates artificial datasets that mimic production properties for unit testing, seeding, and demonstrations. ([source](https://cdn.jsdelivr.net/gh/nucleuscloud/neosync@main/README.md))
- [Relational Database Subsetting](https://awesome-repositories.com/f/artificial-intelligence-ml/dataset-subset-extractions/relational-database-subsetting.md) — Extracts specific data slices from production databases while maintaining referential integrity for local development.

### Security & Cryptography

- [Data Anonymization](https://awesome-repositories.com/f/security-cryptography/data-anonymization.md) — Removes personally identifiable information from production datasets to enable safe local development and testing. ([source](https://cdn.jsdelivr.net/gh/nucleuscloud/neosync@main/README.md))
- [PII Detection and Screening](https://awesome-repositories.com/f/security-cryptography/pii-detection-and-screening.md) — Identifies and masks personally identifiable information using pattern matching and replacement logic to ensure privacy compliance.
- [PII Security Platforms](https://awesome-repositories.com/f/security-cryptography/pii-security-platforms.md) — Offers a comprehensive platform to monitor, detect, and anonymize PII within production datasets for privacy compliance.

### Software Engineering & Architecture

- [Pipeline Orchestration](https://awesome-repositories.com/f/software-engineering-architecture/event-sourcing/pipeline-orchestration.md) — Implements an event-sourced coordinator to ensure reliable data pipeline execution with automated retry and failure handling.
- [Synthetic Dataset Generation](https://awesome-repositories.com/f/software-engineering-architecture/schema-driven-generators/synthetic-dataset-generation.md) — Produces artificial datasets by analyzing database schemas to maintain relational integrity without using real production data.
- [Background Data Synchronization](https://awesome-repositories.com/f/software-engineering-architecture/background-data-synchronization.md) — Provides background workers to move datasets between environments without blocking the main application execution.