github/gh-ost
Gh Ost
gh-ost is a triggerless online schema migration tool for MySQL. It functions as a replication client and table management utility that synchronizes data from a source table to a shadow table using binary logs, allowing for table structure modifications without locking original tables or causing downtime.
The tool distinguishes itself by using binary-log-based replication instead of triggers to stream row-based events to a shadow table. It implements load-aware throttling and dynamic performance tuning to adjust migration speed based on server load and replication lag. Users can monitor and adjust the migration process in real time through a socket-based interactive control interface.
The project covers a broad range of database operation capabilities, including atomic table swapping for zero-downtime cut-overs and parallel execution of multiple schema migrations. It provides verification tools for migration integrity, such as replica simulation, checksum validation, and dry-run migrations. The system also supports various database topologies and includes an event-driven hook system for executing external scripts during the migration lifecycle.
Features
- Schema Migration Utilities - Provides a utility for applying database table structure modifications online without locking tables or using triggers.
- Atomic Table Cut-Overs - Swaps the original table with the migrated version using a safety mechanism to prevent outages.
- Atomic Table Swaps - Implements atomic rename operations to exchange original and migrated tables with minimal downtime.
- Binary Log Streaming Tools - Uses binary log streaming to capture row-based events and apply them to a shadow table without triggers.
- Shadow Table Mirroring - Copies data to a temporary shadow table to perform schema changes without impacting the original table.
- Zero-Downtime Migrations - Enables updating large database tables with new columns or indexes without causing service outages.
- Database Migration Tools - Functions as a database migration tool that synchronizes data from a source table to a shadow table using binary logs.
- Load-Aware Throttling - Adjusts migration speed in real time based on server load and replication lag to prevent performance degradation.
- Migration Resource Throttling - Throttles data transfer speeds and adjusts resource consumption in real time to prevent database server overload.
- Binlog-Based Replication Management - Handles data synchronization and binary log streaming to move table changes from a source to a shadow table.
- Online Schema Migrations - Modifies MySQL table structures without locking the database or using triggers to maintain application availability.
- Online Schema Migration Tools - Provides a triggerless utility for modifying MySQL table structures without locking tables or causing downtime.
- Primary-Replica Replication - Connects to a replica to capture and apply binary log events to the master table.
- Lag Reduction Strategies - Tracks replication lag to dynamically throttle migration speed and reduce performance impact on the server.
- Replication Clients - Acts as a replication client that monitors replication lag and streams row-based events to maintain consistency.
- Load-Aware Throttling - Provides load-aware throttling that adjusts migration speed based on server load and replication lag to prevent performance degradation.
- Data Migration - Copies table data in small segments to avoid long-running transactions and lock contention.
- Concurrent Migration Execution - Executes multiple schema changes simultaneously on a single server using unique process identifiers.
- Interactive Migration Managers - Allows users to send text commands via socket or TCP to monitor progress and adjust configuration live.
- Table Management Utilities - Executes atomic table swaps and manages concurrent schema migrations across master and replica topologies.
- Database Migrations - Cleans up temporary ghost and changelog tables after a crash to reset the database state.
- Database Performance Tuning - Implements dynamic adjustment of chunk sizes and sleep ratios to balance migration speed with database stability.
- Deferred Table Cut-Overs - Pauses the final swap of tables after synchronization to maintain a live mirror until ready.
- Migration Simulations - Simulates the migration process on a replica server to verify data correctness and behavior before updating the master.
- Migration Lifecycle Tools - Provides an interface to suspend operations, cancel throttling, or postpone the final table swap.
- Migration Process Controls - Allows suspending write operations based on replication lag or server load to prevent database overload.
- Topology Configurations - Adapts migration workflows for various setups including master-only, replica-based, and active-passive environments.
- Replica Migration Validations - Executes full schema migrations on replicas to verify data integrity before the final table swap on the master.
- Parallel Migration Execution - Enables the simultaneous execution of multiple schema migrations across different tables to improve efficiency.
- Dry-Run Modes - Simulates the migration process without copying data or flipping tables to verify configuration.
- Data Integrity Checksums - Verifies data integrity via checksums on a replica before applying migration changes to production.
- Unix Socket Interfaces - Provides a control interface via Unix sockets or TCP for real-time configuration and monitoring.
- Database Schema Validations - Tests schema changes on replica servers to verify data integrity before applying updates to the production master.
- Real Time Process Monitors - Provides real-time status updates and progress monitoring via a Unix socket or TCP connection.
- Migration Integrity Verifications - Allows for manual data comparison between source and migrated tables to ensure consistency before the final cut-over.
- Database Tools - Online MySQL schema migration tool.
- Schema Migration Tools - Online schema migration tool for MySQL.
- Development and Schema - Performs online schema migrations without locking tables.
