DTStack/chunjun
Chunjun
Chunjun is a distributed data integration framework and SQL-based ETL pipeline designed to synchronize data between heterogeneous sources. It functions as a change data capture tool and a heterogeneous data synchronizer, utilizing a distributed processing environment to move and transform data across different database types.
The system is distinguished by its plugin-based connector architecture, which allows for the development of custom source and sink plugins to extend connectivity to unsupported data systems. It supports real-time change data capture from relational database logs and implements schema evolution propagation to automatically apply structural changes from source to destination tables.
The framework provides capabilities for incremental data synchronization and cross-source data calculation using SQL logic. Reliability is managed through checkpoint-based task recovery to resume interrupted transfers and dead-letter queues for dirty data management to audit malformed records.
Integration tasks can be deployed across standalone clusters, Yarn, or Kubernetes environments, with support for containerized deployment via Docker.
Features
- Distributed Data Processing Frameworks - Provides a distributed framework for synchronizing and transforming data between heterogeneous sources using a plugin-based architecture.
- Heterogeneous Data Synchronization - Transfers and aligns data between different heterogeneous data sources using a distributed integration framework.
- Change Data Capture - Streams real-time updates from relational database logs to enable low-latency synchronization between heterogeneous systems.
- Change Data Capture Tools - Collects data from relational databases in real-time via logs to facilitate low-latency synchronization.
- Checkpoints and Recovery - Resumes interrupted data transfers from the last successful checkpoint to ensure disaster recovery and data consistency.
- Distributed Cluster Execution - Spreads data integration workloads across multiple nodes using Yarn or Kubernetes for parallel processing.
- Incremental Data Synchronization - Transfers only new or changed data records over time instead of performing full dataset copies.
- SQL-Based Pipeline Definitions - Allows defining data movement and transformation workflows using SQL declarations and JSON templates.
- Data Pipeline Deployments - Enables the deployment of large-scale data movement tasks across Kubernetes, Yarn, or standalone clusters.
- Plugin-Based Architectures - Provides a plugin-based connector architecture with standardized read and write interfaces for heterogeneous data sources and sinks.
- Connector Plugin Development - The product allows developers to create new source or sink connectors to synchronize data between heterogeneous systems by implementing read and write logic.
- Cross-Source Data Integration - Joins and calculates data between diverse sources using a plugin-based architecture to ensure cross-database compatibility.
- Data Quality Monitors - Captures failing records and provides metrics to monitor overall data quality during the synchronization process.
- Dirty Data Capture - Isolates and stores malformed records that fail processing to prevent pipeline crashes and enable correction.
- Incremental Sync Checkpointings - Monitors data sources and utilizes checkpoint-based resume to ensure consistency during incremental transfers.
- Distributed SQL Computations - Performs data computation and transformation tasks using SQL logic within a distributed processing environment.
- Schema Synchronizers - Aligns structural definitions of source and destination tables to maintain data integrity across heterogeneous systems.
- Automated Schema Propagation - Automatically propagates structural changes from source databases to destination tables.
- SQL-Based CDC Integrations - Enables the definition of data integration and CDC workflows using SQL scripts compatible with streaming syntax.
- Data Integration Task Definitions - Allows defining data movement jobs and source-to-destination mappings using JSON or SQL declarations.
- Error Tracking and Exception Handling - Provides a dead-letter queue to capture and track malformed records that fail during synchronization for later auditing.
- Data Processing Orchestrators - Orchestrates data processing pipelines as scalable jobs within Kubernetes, Yarn, or standalone environments.
- Data Source Extensions - Provides mechanisms to extend connectivity to unsupported data systems via custom reader, writer, and lookup plugins.
- Checkpoint-Based Resumptions - Implements mechanisms to save data offsets, allowing interrupted synchronization tasks to resume from the last successful checkpoint.
- Declarative Configuration Systems - Allows defining data movement workflows and processing pipelines using declarative JSON or SQL scripts.
- Dead Letter Queues - Utilizes dead-letter queues to isolate and store malformed records for auditing and manual correction.
Star history
How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
