zendesk/maxwell

Features

• MySQL Change Data Capture - Reading MySQL binlogs to turn database modifications into JSON events for streaming to other systems.
• Change Data Capture - Provides a system for streaming row-level database inserts, updates, and deletes as a sequence of JSON events.
• Change Data Capture - Writes captured database changes as formatted JSON data to external destinations for downstream processing.
• Change Data Capture Tools - Reads MySQL binary logs and converts database modifications into JSON events for streaming platforms.
• Change Filters - Provides rules for including or excluding specific tables, columns, or rows from the change stream using regex or name rules.
• Snapshot Bootstrapping - Captures a point-in-time copy of existing table data to seed the stream before real-time capture begins.
• Database-to-Kafka Connectors - Streams real-time database changes from MySQL into Kafka and other distributed messaging systems.
• Partition-Based Routing - Distributes events across streaming partitions using keys to ensure strict chronological ordering of database updates.
• Binlog Streaming Proxies - Monitors MySQL binary logs to synchronize data changes across external indices and search engines.
• Snapshot-Based Bootstrapping - Capture a snapshot of existing table data and stream it as initial events before starting real-time capture.
• Real-time Data Synchronization - Keeps external search indices and secondary databases consistent by streaming live updates from a primary MySQL store.
• Stream Processing Pipelines - Implements a high-throughput data pipeline that feeds database changes into streaming platforms with batching and partitioning.
• Database Audit Trails - Captures a chronological history of all database modifications to maintain persistent audit trails.
• Log Position Recovery - Tracks and restores the last processed binary log position to ensure continuous data capture after a crash or failover.
• Messaging Platform Integrations - Integrates with messaging platforms to route captured database events for inter-service communication and transformation.
• Database Change Streaming - Reads database binary logs and streams real-time row and column updates to external platforms.
• Service-to-Service Streaming - Sends database change events to message brokers to trigger workflows and notify other microservices.
• Entity Audit Logs - Creates a persistent and chronological history of every database modification for compliance and tracking.
• Event-Driven Architectures - Routes database changes to Kafka or Kinesis to trigger workflows and notify microservices.
• Batch Write Buffering - Implements batch write buffering to increase throughput and reduce network overhead when streaming database changes.
• Database Record Auditing - Records every change made to a database, tracking mutations of individual records with old and new values.
• Search Index Synchronization - Streams database change logs to external search engines in real-time to maintain consistent search indices.
• Schema Change Logs - Monitors table and column modifications as events and records them in schema change logs.
• Schema Version Tracking - Tracks database schema changes and manages version history to maintain an accurate structural record.
• Schema History Compaction - Tracks database structure changes over time and rolls up old versions to limit storage growth.
• Dynamic Configuration Hot-Loading - Modify operational settings via an HTTP endpoint to change behavior without restarting the application.
• Binlog Position Recovery - Restores the log reading position when a slave is promoted to master to ensure continuous data capture.
• Batch-Buffered Stream Writing - Pack multiple small records into a single larger batch to increase throughput and prevent throttling during stream writes.
• Structured JSON Loggers - Transforms raw database change logs into structured JSON records for downstream processing and storage.
• Hot-Reload Configuration Managers - Provides a management interface for applying operational configuration changes at runtime without restarting the application.
• Change Data Capture - Tool that streams MySQL binlog changes as JSON to Kafka.

Star history