alibaba/DataX
DataX
DataX is a distributed data integration framework and plugin-based ETL tool designed for synchronizing large datasets between heterogeneous sources and destinations. It functions as a JDBC data migration engine and offline synchronization tool, enabling the movement of data between relational databases, NoSQL stores, and object storage.
The system utilizes a plugin-based connector architecture that decouples reader and writer logic, allowing it to map and transform data types across different storage engines using a standardized internal representation. This design supports heterogeneous data pipelines where source-specific data is converted into compatible target types to ensure cross-platform compatibility.
The framework provides comprehensive capabilities for data extraction, including support for columnar formats, incremental synchronization via SQL filtering, and archive decompression. Its writing surface includes batch commit operations, idempotent write strategies to maintain consistency during retries, and the ability to execute pre- and post-synchronization SQL scripts.
Performance is managed through task-level parallelism, throughput control to regulate memory and network traffic, and batch-based write buffering to increase ingestion speed.
Features
- Data Extraction - Provides core tools for isolating and retrieving specific data points from relational database sources.
- Large Scale Data Integration Frameworks - Functions as a distributed framework for synchronizing massive volumes of data between heterogeneous sources and destinations.
- Plugin-Based Architectures - Built on a plugin-based architecture that decouples reader and writer logic to support heterogeneous system integration.
- Analytical Data Loads from Object Storage - Loads data from cloud object storage into a transportable format for analytical processing.
- Relational Data Synchronization - Synchronizes relational database records from sources like MySQL, PostgreSQL, and Oracle into target tables via JDBC.
- Cross-Database Data Migrations - Migrates large datasets between disparate database engines using a standardized internal representation for cross-platform compatibility.
- Database-Specific Extractions - Enables record extraction from OceanBase databases via JDBC and SQL queries for migration.
- Data Insertion Interfaces - Implements a programmatic interface for inserting records into target systems via standard JDBC statements.
- Data Integration Pipelines - Orchestrates the movement and routing of data between object storage, graph databases, and analytical warehouses via a plugin architecture.
- Intermediate Representations - Employs internal data models that normalize diverse input formats into a consistent structure for uniform processing across different storage engines.
- Data Type Mappings - Translates source data types into compatible target database-specific column formats.
- Data Extraction - Implements specialized extraction logic to read records from MySQL databases using JDBC and SQL queries.
- Data Extraction - Retrieves records from remote Oracle databases using JDBC and generated or custom SQL statements.
- Distributed Batch Processing - Transfers terabyte-scale datasets using parallel extraction and distributed writes to maximize system throughput.
- Heterogeneous Data Pipelines - Implements pipelines that transform and map data types across different storage engines to ensure cross-platform compatibility.
- Heterogeneous Data Synchronization - Enables data migration between diverse storage types such as relational databases and NoSQL stores using a standardized internal format.
- Incremental Data Synchronization - Synchronizes only new or modified records by filtering data using WHERE clauses based on timestamps or IDs.
- JDBC Migration Engines - Leverages JDBC drivers to extract and load records across a wide variety of relational database management systems.
- Plugin-Based ETL Frameworks - Uses a plugin-based connector architecture to decouple reader and writer logic, allowing extensions for new heterogeneous data sources.
- Pre and Post Load SQL Execution - Executes custom SQL statements immediately before or after synchronization tasks to prepare or finalize data.
- Relational Database Writers - Inserts data into target relational database tables using JDBC connections and custom drivers.
- Data Extraction - Reads records from remote SQL Server databases using JDBC connections and SQL SELECT statements.
- Data Extraction - Extracts data from remote PostgreSQL databases using JDBC connections and SQL select statements.
- Data Partition Parallelism - Supports task-level parallelism by splitting large datasets into independent chunks for concurrent extraction across threads.
- Hub-and-Spoke Data Flows - Utilizes a hub-and-spoke data flow to route information from specialized readers through a central engine to specialized writers.
- Data Component Plugins - Integrates new data sources through the implementation of customizable reader and writer plugins.
- Parallel Data Pipelines - Splits large datasets into concurrent tasks based on primary keys to increase synchronization speed.
- Object Selection Filters - Supports isolating specific objects for extraction through name-based filtering and directory traversal.
- Batch Write Buffering - Groups multiple record writes into a single transaction to increase data ingestion speed and reduce network overhead.
- Custom SQL Execution - Allows the use of custom SQL queries for data extraction to enable complex operations like multi-table joins.
- Export Throughput Limiters - Regulates memory usage and network traffic by capping batch sizes and thread counts during data import.
- Data Synchronization Tools - Provides a tool for bulk data migrations and incremental synchronizations between relational databases and NoSQL stores.
- Data Transformation Functions - Provides built-in functions and custom rules for masking, completing, and filtering data during the migration process.
- Unstructured Text Converters - Translates raw string values from object storage into structured types such as decimals and integers.
- Dirty Data Captures - Provides a dirty-data capture mechanism to intercept and isolate records failing type conversion, ensuring pipeline stability.
- Graph Database Writers - Synchronizes data into Neo4j using Cypher queries to create nodes and relationships.
- Graph Database Exporters - Exports data into graph databases by converting source records into vertices and edges.
- Idempotent Write Strategies - Implements idempotent write strategies that clear target partitions before writing to maintain consistency during failed task retries.
- Bulk Load Optimizations - Moves terabyte-scale data by leveraging temporary distributed storage before triggering optimized bulk load commands.
- Columnar Tabular Storage - Extracts data from optimized columnar storage files including Parquet and ORC for large-scale processing.
- NoSQL Data Writers - Transfers structured data into OTS NoSQL databases with multi-version record support.
- Parallel Storage Writing - Distributes data across multiple threads to write different sub-files simultaneously, improving overall ingestion throughput.
- Relational Database Drivers - Integrates various relational database types by registering JDBC drivers and adding necessary driver files.
- Schema Column Mapping - Selects specific columns for import and rearranges their order to align with the destination schema.
- SQL Data Retrieval - Implements techniques for filtering and extracting specific data from relational tables using SQL WHERE clauses.
- Structured Data File Extractors - Extracts text and field names from structured data files such as CSV and TXT using custom delimiters.
- Tabular Object Storage - Transfers tabular data into object storage using structured formats like Parquet, ORC, and CSV.
- Throughput Controls - Provides mechanisms to cap transfer rates using concurrency channels or byte limits to prevent target system overloading.
- Column Projection - Prunes data during export by selecting a specific subset of columns and defining their output order.
- Throughput Controllers - Regulates memory usage and network traffic by capping concurrency levels and batch sizes during the transfer process.
- Database Batch Writes - Implements batch-based write buffering to group records into single transactions, reducing network overhead and increasing ingestion speed.
- Transfer Retry Mechanisms - Automatically retries failed operations at the thread or task level to ensure completion during network instability.
- Data Exchange and ETL - Data synchronization and exchange tool.
