DTStack/chunjun
Chunjun
Chunjun est un framework d'intégration de données distribué et un pipeline ETL basé sur SQL conçu pour synchroniser les données entre des sources hétérogènes. Il fonctionne comme un outil de capture de données modifiées (CDC) et un synchroniseur de données hétérogènes, utilisant un environnement de traitement distribué pour déplacer et transformer les données à travers différents types de bases de données.
Le système se distingue par son architecture de connecteurs basée sur des plugins, qui permet le développement de plugins de source et de destination personnalisés pour étendre la connectivité aux systèmes de données non pris en charge. Il prend en charge la capture de données modifiées en temps réel à partir des journaux de bases de données relationnelles et implémente la propagation de l'évolution de schéma pour appliquer automatiquement les changements structurels des tables sources aux tables de destination.
Le framework fournit des capacités pour la synchronisation de données incrémentielle et le calcul de données inter-sources utilisant la logique SQL. La fiabilité est gérée via une récupération de tâche basée sur des points de contrôle pour reprendre les transferts interrompus et des files d'attente de lettres mortes pour la gestion des données sales afin d'auditer les enregistrements mal formés.
Les tâches d'intégration peuvent être déployées sur des clusters autonomes, Yarn ou des environnements Kubernetes, avec une prise en charge du déploiement conteneurisé via Docker.
Features
- Distributed Data Processing Frameworks - Provides a distributed framework for synchronizing and transforming data between heterogeneous sources using a plugin-based architecture.
- Heterogeneous Data Synchronization - Transfers and aligns data between different heterogeneous data sources using a distributed integration framework.
- Change Data Capture - Streams real-time updates from relational database logs to enable low-latency synchronization between heterogeneous systems.
- Change Data Capture Tools - Collects data from relational databases in real-time via logs to facilitate low-latency synchronization.
- Checkpoints and Recovery - Resumes interrupted data transfers from the last successful checkpoint to ensure disaster recovery and data consistency.
- Distributed Cluster Execution - Spreads data integration workloads across multiple nodes using Yarn or Kubernetes for parallel processing.
- Incremental Data Synchronization - Transfers only new or changed data records over time instead of performing full dataset copies.
- SQL-Based Pipeline Definitions - Allows defining data movement and transformation workflows using SQL declarations and JSON templates.
- Data Pipeline Deployments - Enables the deployment of large-scale data movement tasks across Kubernetes, Yarn, or standalone clusters.
- Plugin-Based Architectures - Provides a plugin-based connector architecture with standardized read and write interfaces for heterogeneous data sources and sinks.
- Connector Plugin Development - The product allows developers to create new source or sink connectors to synchronize data between heterogeneous systems by implementing read and write logic.
- Cross-Source Data Integration - Joins and calculates data between diverse sources using a plugin-based architecture to ensure cross-database compatibility.
- Data Quality Monitors - Captures failing records and provides metrics to monitor overall data quality during the synchronization process.
- Dirty Data Capture - Isolates and stores malformed records that fail processing to prevent pipeline crashes and enable correction.
- Incremental Sync Checkpointings - Monitors data sources and utilizes checkpoint-based resume to ensure consistency during incremental transfers.
- Distributed SQL Computations - Performs data computation and transformation tasks using SQL logic within a distributed processing environment.
- Schema Synchronizers - Aligns structural definitions of source and destination tables to maintain data integrity across heterogeneous systems.
- Automated Schema Propagation - Automatically propagates structural changes from source databases to destination tables.
- SQL-Based CDC Integrations - Enables the definition of data integration and CDC workflows using SQL scripts compatible with streaming syntax.
- Data Integration Task Definitions - Allows defining data movement jobs and source-to-destination mappings using JSON or SQL declarations.
- Error Tracking and Exception Handling - Provides a dead-letter queue to capture and track malformed records that fail during synchronization for later auditing.
- Data Processing Orchestrators - Orchestrates data processing pipelines as scalable jobs within Kubernetes, Yarn, or standalone environments.
- Data Source Extensions - Provides mechanisms to extend connectivity to unsupported data systems via custom reader, writer, and lookup plugins.
- Checkpoint-Based Resumptions - Implements mechanisms to save data offsets, allowing interrupted synchronization tasks to resume from the last successful checkpoint.
- Declarative Configuration Systems - Allows defining data movement workflows and processing pipelines using declarative JSON or SQL scripts.
- Dead Letter Queues - Utilizes dead-letter queues to isolate and store malformed records for auditing and manual correction.
