Replibyte

Replibyte is a tool that automates the lifecycle of database snapshots for non-production environments, handling the export, anonymization, subsetting, and restoration of data. It is designed to support privacy-compliant development workflows by replacing sensitive production data with synthetic values and extracting consistent subsets of rows while preserving referential integrity.

The tool operates through a configurable pipeline defined in a YAML file, orchestrating stages such as dump, anonymize, subset, and restore. Each operation runs as an isolated, ephemeral container job, and snapshots are stored as encrypted files in remote object storage services like S3 or GCS. Replibyte also manages snapshot retention by automatically removing dumps based on age or count, and it can seed development databases with realistic, anonymized production data.

The project provides a command-line interface for configuring and triggering these operations, with support for running as a lifecycle job within deployment environments.

Features

Relational Database Subsetting - Extracts a configurable percentage of rows while following foreign-key relationships to maintain referential integrity.

Encrypted Snapshots - Stores database dumps as encrypted snapshots in remote object storage for secure data at rest.

Encrypted Snapshot Dumpers - Creates encrypted snapshots of databases and stores them in remote cloud object storage.

Non-Production Data Pipelines - Automates export, anonymization, and restoration of database content for non-production environments.

Data Seeding Utilities - Restores production database snapshots into development environments for realistic testing.

Production Snapshot Seeding - Restores anonymized production snapshots into non-production environments for realistic development testing.

Production Data Subsetting - Extracts a configurable percentage of rows from database tables while preserving referential integrity.

Database Snapshots - Automates dumping, encrypted storage, and lifecycle cleanup of database snapshots in cloud object storage.

Data Anonymization - Replaces sensitive production data with synthetic values to meet privacy regulations before moving data to non-production environments.

Column-Level Anonymization - Automatically replaces specified columns with anonymized values during the restore process to protect private information.

Database Anonymization Tools - Replaces sensitive production data with synthetic values during database restores to ensure privacy compliance.

Configuration Workflows - Defines the entire data lifecycle pipeline through a YAML configuration file for repeatable operations.

Database Lifecycle Pipelines - Orchestrates database operations as a sequence of dump, anonymize, subset, and restore stages.

Production Data Seeding - Restores realistic, anonymized production snapshots into development or staging databases for accurate testing.

Anonymization Rules - Replaces sensitive database column values with synthetic data during restore for privacy compliance.

Remote Object Storage Integrations - Transfers and retrieves database snapshots directly from cloud object storage services like S3 or GCS.

Ephemeral Job Runners - Executes each database lifecycle stage as an isolated, ephemeral container job.

Backup Lifecycle Automation - Automates retention policies by removing old or excess database dumps based on age or count.

QoveryReplibyte

Features

Star history