DTStack/chunjun

Features

• Distributed Data Processing Frameworks - Provides a distributed framework for synchronizing and transforming data between heterogeneous sources using a plugin-based architecture.
• Heterogeneous Data Synchronization - Transfers and aligns data between different heterogeneous data sources using a distributed integration framework.
• Change Data Capture - Streams real-time updates from relational database logs to enable low-latency synchronization between heterogeneous systems.
• Change Data Capture Tools - Collects data from relational databases in real-time via logs to facilitate low-latency synchronization.
• Checkpoints and Recovery - Resumes interrupted data transfers from the last successful checkpoint to ensure disaster recovery and data consistency.
• Distributed Cluster Execution - Spreads data integration workloads across multiple nodes using Yarn or Kubernetes for parallel processing.
• Incremental Data Synchronization - Transfers only new or changed data records over time instead of performing full dataset copies.
• SQL-Based Pipeline Definitions - Allows defining data movement and transformation workflows using SQL declarations and JSON templates.
• Data Pipeline Deployments - Enables the deployment of large-scale data movement tasks across Kubernetes, Yarn, or standalone clusters.
• Plugin-Based Architectures - Provides a plugin-based connector architecture with standardized read and write interfaces for heterogeneous data sources and sinks.
• Connector Plugin Development - The product allows developers to create new source or sink connectors to synchronize data between heterogeneous systems by implementing read and write logic.
• Cross-Source Data Integration - Joins and calculates data between diverse sources using a plugin-based architecture to ensure cross-database compatibility.
• Data Quality Monitors - Captures failing records and provides metrics to monitor overall data quality during the synchronization process.
• Dirty Data Capture - Isolates and stores malformed records that fail processing to prevent pipeline crashes and enable correction.
• Incremental Sync Checkpointings - Monitors data sources and utilizes checkpoint-based resume to ensure consistency during incremental transfers.
• Distributed SQL Computations - Performs data computation and transformation tasks using SQL logic within a distributed processing environment.
• Schema Synchronizers - Aligns structural definitions of source and destination tables to maintain data integrity across heterogeneous systems.
• Automated Schema Propagation - Automatically propagates structural changes from source databases to destination tables.
• SQL-Based CDC Integrations - Enables the definition of data integration and CDC workflows using SQL scripts compatible with streaming syntax.
• Data Integration Task Definitions - Allows defining data movement jobs and source-to-destination mappings using JSON or SQL declarations.
• Error Tracking and Exception Handling - Provides a dead-letter queue to capture and track malformed records that fail during synchronization for later auditing.
• Data Processing Orchestrators - Orchestrates data processing pipelines as scalable jobs within Kubernetes, Yarn, or standalone environments.
• Data Source Extensions - Provides mechanisms to extend connectivity to unsupported data systems via custom reader, writer, and lookup plugins.
• Checkpoint-Based Resumptions - Implements mechanisms to save data offsets, allowing interrupted synchronization tasks to resume from the last successful checkpoint.
• Declarative Configuration Systems - Allows defining data movement workflows and processing pipelines using declarative JSON or SQL scripts.
• Dead Letter Queues - Utilizes dead-letter queues to isolate and store malformed records for auditing and manual correction.

Star 历史