DTStack/chunjun
Chunjun
Chunjun este un framework distribuit de integrare a datelor și pipeline ETL bazat pe SQL, conceput pentru a sincroniza datele între surse eterogene. Acesta funcționează ca un instrument de change data capture și un sincronizator de date eterogene, utilizând un mediu de procesare distribuit pentru a muta și transforma datele între diferite tipuri de baze de date.
Sistemul se distinge prin arhitectura sa de conectori bazată pe plugin-uri, care permite dezvoltarea de plugin-uri personalizate de sursă și destinație pentru a extinde conectivitatea către sisteme de date neacceptate. Suportă change data capture în timp real din log-urile bazelor de date relaționale și implementează propagarea evoluției schemei pentru a aplica automat modificările structurale de la tabelele sursă la cele de destinație.
Framework-ul oferă capabilități pentru sincronizarea incrementală a datelor și calculul datelor între surse folosind logica SQL. Fiabilitatea este gestionată prin recuperarea sarcinilor bazată pe checkpoint-uri pentru a relua transferurile întrerupte și cozi de mesaje dead-letter pentru gestionarea datelor murdare, pentru a audita înregistrările malformate.
Sarcinile de integrare pot fi implementate pe clustere standalone, Yarn sau medii Kubernetes, cu suport pentru implementare containerizată prin Docker.
Features
- Distributed Data Processing Frameworks - Provides a distributed framework for synchronizing and transforming data between heterogeneous sources using a plugin-based architecture.
- Heterogeneous Data Synchronization - Transfers and aligns data between different heterogeneous data sources using a distributed integration framework.
- Change Data Capture - Streams real-time updates from relational database logs to enable low-latency synchronization between heterogeneous systems.
- Change Data Capture Tools - Collects data from relational databases in real-time via logs to facilitate low-latency synchronization.
- Checkpoints and Recovery - Resumes interrupted data transfers from the last successful checkpoint to ensure disaster recovery and data consistency.
- Distributed Cluster Execution - Spreads data integration workloads across multiple nodes using Yarn or Kubernetes for parallel processing.
- Incremental Data Synchronization - Transfers only new or changed data records over time instead of performing full dataset copies.
- SQL-Based Pipeline Definitions - Allows defining data movement and transformation workflows using SQL declarations and JSON templates.
- Data Pipeline Deployments - Enables the deployment of large-scale data movement tasks across Kubernetes, Yarn, or standalone clusters.
- Plugin-Based Architectures - Provides a plugin-based connector architecture with standardized read and write interfaces for heterogeneous data sources and sinks.
- Connector Plugin Development - The product allows developers to create new source or sink connectors to synchronize data between heterogeneous systems by implementing read and write logic.
- Cross-Source Data Integration - Joins and calculates data between diverse sources using a plugin-based architecture to ensure cross-database compatibility.
- Data Quality Monitors - Captures failing records and provides metrics to monitor overall data quality during the synchronization process.
- Dirty Data Capture - Isolates and stores malformed records that fail processing to prevent pipeline crashes and enable correction.
- Incremental Sync Checkpointings - Monitors data sources and utilizes checkpoint-based resume to ensure consistency during incremental transfers.
- Distributed SQL Computations - Performs data computation and transformation tasks using SQL logic within a distributed processing environment.
- Schema Synchronizers - Aligns structural definitions of source and destination tables to maintain data integrity across heterogeneous systems.
- Automated Schema Propagation - Automatically propagates structural changes from source databases to destination tables.
- SQL-Based CDC Integrations - Enables the definition of data integration and CDC workflows using SQL scripts compatible with streaming syntax.
- Data Integration Task Definitions - Allows defining data movement jobs and source-to-destination mappings using JSON or SQL declarations.
- Error Tracking and Exception Handling - Provides a dead-letter queue to capture and track malformed records that fail during synchronization for later auditing.
- Data Processing Orchestrators - Orchestrates data processing pipelines as scalable jobs within Kubernetes, Yarn, or standalone environments.
- Data Source Extensions - Provides mechanisms to extend connectivity to unsupported data systems via custom reader, writer, and lookup plugins.
- Checkpoint-Based Resumptions - Implements mechanisms to save data offsets, allowing interrupted synchronization tasks to resume from the last successful checkpoint.
- Declarative Configuration Systems - Allows defining data movement workflows and processing pipelines using declarative JSON or SQL scripts.
- Dead Letter Queues - Utilizes dead-letter queues to isolate and store malformed records for auditing and manual correction.
