DTStack/chunjun
Chunjun
Chunjun es un framework de integración de datos distribuido y pipeline ETL basado en SQL diseñado para sincronizar datos entre fuentes heterogéneas. Funciona como una herramienta de captura de datos modificados (CDC) y un sincronizador de datos heterogéneos, utilizando un entorno de procesamiento distribuido para mover y transformar datos a través de diferentes tipos de bases de datos.
El sistema se distingue por su arquitectura de conectores basada en plugins, que permite el desarrollo de plugins de origen y destino personalizados para extender la conectividad a sistemas de datos no soportados. Admite la captura de datos modificados en tiempo real desde registros de bases de datos relacionales e implementa la propagación de evolución de esquemas para aplicar automáticamente cambios estructurales de tablas de origen a destino.
El framework proporciona capacidades para la sincronización de datos incremental y el cálculo de datos entre fuentes utilizando lógica SQL. La fiabilidad se gestiona a través de la recuperación de tareas basada en puntos de control para reanudar transferencias interrumpidas y colas de mensajes fallidos (dead-letter queues) para la gestión de datos sucios para auditar registros malformados.
Las tareas de integración pueden desplegarse a través de clusters independientes, Yarn o entornos Kubernetes, con soporte para despliegue contenedorizado a través de Docker.
Features
- Distributed Data Processing Frameworks - Provides a distributed framework for synchronizing and transforming data between heterogeneous sources using a plugin-based architecture.
- Heterogeneous Data Synchronization - Transfers and aligns data between different heterogeneous data sources using a distributed integration framework.
- Change Data Capture - Streams real-time updates from relational database logs to enable low-latency synchronization between heterogeneous systems.
- Change Data Capture Tools - Collects data from relational databases in real-time via logs to facilitate low-latency synchronization.
- Checkpoints and Recovery - Resumes interrupted data transfers from the last successful checkpoint to ensure disaster recovery and data consistency.
- Distributed Cluster Execution - Spreads data integration workloads across multiple nodes using Yarn or Kubernetes for parallel processing.
- Incremental Data Synchronization - Transfers only new or changed data records over time instead of performing full dataset copies.
- SQL-Based Pipeline Definitions - Allows defining data movement and transformation workflows using SQL declarations and JSON templates.
- Data Pipeline Deployments - Enables the deployment of large-scale data movement tasks across Kubernetes, Yarn, or standalone clusters.
- Plugin-Based Architectures - Provides a plugin-based connector architecture with standardized read and write interfaces for heterogeneous data sources and sinks.
- Connector Plugin Development - The product allows developers to create new source or sink connectors to synchronize data between heterogeneous systems by implementing read and write logic.
- Cross-Source Data Integration - Joins and calculates data between diverse sources using a plugin-based architecture to ensure cross-database compatibility.
- Data Quality Monitors - Captures failing records and provides metrics to monitor overall data quality during the synchronization process.
- Dirty Data Capture - Isolates and stores malformed records that fail processing to prevent pipeline crashes and enable correction.
- Incremental Sync Checkpointings - Monitors data sources and utilizes checkpoint-based resume to ensure consistency during incremental transfers.
- Distributed SQL Computations - Performs data computation and transformation tasks using SQL logic within a distributed processing environment.
- Schema Synchronizers - Aligns structural definitions of source and destination tables to maintain data integrity across heterogeneous systems.
- Automated Schema Propagation - Automatically propagates structural changes from source databases to destination tables.
- SQL-Based CDC Integrations - Enables the definition of data integration and CDC workflows using SQL scripts compatible with streaming syntax.
- Data Integration Task Definitions - Allows defining data movement jobs and source-to-destination mappings using JSON or SQL declarations.
- Error Tracking and Exception Handling - Provides a dead-letter queue to capture and track malformed records that fail during synchronization for later auditing.
- Data Processing Orchestrators - Orchestrates data processing pipelines as scalable jobs within Kubernetes, Yarn, or standalone environments.
- Data Source Extensions - Provides mechanisms to extend connectivity to unsupported data systems via custom reader, writer, and lookup plugins.
- Checkpoint-Based Resumptions - Implements mechanisms to save data offsets, allowing interrupted synchronization tasks to resume from the last successful checkpoint.
- Declarative Configuration Systems - Allows defining data movement workflows and processing pipelines using declarative JSON or SQL scripts.
- Dead Letter Queues - Utilizes dead-letter queues to isolate and store malformed records for auditing and manual correction.
