zendesk/maxwell
Maxwell
Maxwell is a MySQL change data capture tool and binlog streaming application that converts database modifications into structured JSON events. It functions as a data pipeline that reads MySQL binary logs to synchronize changes across external indices, search engines, and distributed messaging systems such as Kafka.
The project provides capabilities to maintain persistent audit trails by recording a chronological history of all database modifications. It enables real-time data synchronization and event-driven architecture integration by streaming database changes to external platforms to trigger workflows and notify microservices.
The system covers broad functional areas including data bootstrapping through initial snapshots, schema version management, and event filtering. It incorporates traffic management through partition-key routing and provides monitoring via health checks and performance metrics exposed through an HTTP endpoint.
Connections to databases and streaming producers are secured using SSL and encrypted communication.
Features
- MySQL Change Data Capture - Reading MySQL binlogs to turn database modifications into JSON events for streaming to other systems.
- Change Data Capture - Provides a system for streaming row-level database inserts, updates, and deletes as a sequence of JSON events.
- Change Data Capture - Writes captured database changes as formatted JSON data to external destinations for downstream processing.
- Change Data Capture Tools - Reads MySQL binary logs and converts database modifications into JSON events for streaming platforms.
- Change Filters - Provides rules for including or excluding specific tables, columns, or rows from the change stream using regex or name rules.
- Snapshot Bootstrapping - Captures a point-in-time copy of existing table data to seed the stream before real-time capture begins.
- Database-to-Kafka Connectors - Streams real-time database changes from MySQL into Kafka and other distributed messaging systems.
- Partition-Based Routing - Distributes events across streaming partitions using keys to ensure strict chronological ordering of database updates.
- Binlog Streaming Proxies - Monitors MySQL binary logs to synchronize data changes across external indices and search engines.
- Snapshot-Based Bootstrapping - Capture a snapshot of existing table data and stream it as initial events before starting real-time capture.
- Real-time Data Synchronization - Keeps external search indices and secondary databases consistent by streaming live updates from a primary MySQL store.
- Stream Processing Pipelines - Implements a high-throughput data pipeline that feeds database changes into streaming platforms with batching and partitioning.
- Database Audit Trails - Captures a chronological history of all database modifications to maintain persistent audit trails.
- Log Position Recovery - Tracks and restores the last processed binary log position to ensure continuous data capture after a crash or failover.
- Messaging Platform Integrations - Integrates with messaging platforms to route captured database events for inter-service communication and transformation.
- Database Change Streaming - Reads database binary logs and streams real-time row and column updates to external platforms.
- Service-to-Service Streaming - Sends database change events to message brokers to trigger workflows and notify other microservices.
- Entity Audit Logs - Creates a persistent and chronological history of every database modification for compliance and tracking.
- Event-Driven Architectures - Routes database changes to Kafka or Kinesis to trigger workflows and notify microservices.
- Batch Write Buffering - Implements batch write buffering to increase throughput and reduce network overhead when streaming database changes.
- Database Record Auditing - Records every change made to a database, tracking mutations of individual records with old and new values.
- Search Index Synchronization - Streams database change logs to external search engines in real-time to maintain consistent search indices.
- Schema Change Logs - Monitors table and column modifications as events and records them in schema change logs.
- Schema Version Tracking - Tracks database schema changes and manages version history to maintain an accurate structural record.
- Schema History Compaction - Tracks database structure changes over time and rolls up old versions to limit storage growth.
- Dynamic Configuration Hot-Loading - Modify operational settings via an HTTP endpoint to change behavior without restarting the application.
- Binlog Position Recovery - Restores the log reading position when a slave is promoted to master to ensure continuous data capture.
- Batch-Buffered Stream Writing - Pack multiple small records into a single larger batch to increase throughput and prevent throttling during stream writes.
- Structured JSON Loggers - Transforms raw database change logs into structured JSON records for downstream processing and storage.
- Hot-Reload Configuration Managers - Provides a management interface for applying operational configuration changes at runtime without restarting the application.
- Change Data Capture - Tool that streams MySQL binlog changes as JSON to Kafka.
