alibaba/DataX

Features

• Data Extraction - Provides core tools for isolating and retrieving specific data points from relational database sources.
• Large Scale Data Integration Frameworks - Functions as a distributed framework for synchronizing massive volumes of data between heterogeneous sources and destinations.
• Plugin-Based Architectures - Built on a plugin-based architecture that decouples reader and writer logic to support heterogeneous system integration.
• Analytical Data Loads from Object Storage - Loads data from cloud object storage into a transportable format for analytical processing.
• Relational Data Synchronization - Synchronizes relational database records from sources like MySQL, PostgreSQL, and Oracle into target tables via JDBC.
• Cross-Database Data Migrations - Migrates large datasets between disparate database engines using a standardized internal representation for cross-platform compatibility.
• Database-Specific Extractions - Enables record extraction from OceanBase databases via JDBC and SQL queries for migration.
• Data Insertion Interfaces - Implements a programmatic interface for inserting records into target systems via standard JDBC statements.
• Data Integration Pipelines - Orchestrates the movement and routing of data between object storage, graph databases, and analytical warehouses via a plugin architecture.
• Intermediate Representations - Employs internal data models that normalize diverse input formats into a consistent structure for uniform processing across different storage engines.
• Data Type Mappings - Translates source data types into compatible target database-specific column formats.
• Data Extraction - Implements specialized extraction logic to read records from MySQL databases using JDBC and SQL queries.
• Data Extraction - Retrieves records from remote Oracle databases using JDBC and generated or custom SQL statements.
• Distributed Batch Processing - Transfers terabyte-scale datasets using parallel extraction and distributed writes to maximize system throughput.
• Heterogeneous Data Pipelines - Implements pipelines that transform and map data types across different storage engines to ensure cross-platform compatibility.
• Heterogeneous Data Synchronization - Enables data migration between diverse storage types such as relational databases and NoSQL stores using a standardized internal format.
• Incremental Data Synchronization - Synchronizes only new or modified records by filtering data using WHERE clauses based on timestamps or IDs.
• JDBC Migration Engines - Leverages JDBC drivers to extract and load records across a wide variety of relational database management systems.
• Plugin-Based ETL Frameworks - Uses a plugin-based connector architecture to decouple reader and writer logic, allowing extensions for new heterogeneous data sources.
• Pre and Post Load SQL Execution - Executes custom SQL statements immediately before or after synchronization tasks to prepare or finalize data.
• Relational Database Writers - Inserts data into target relational database tables using JDBC connections and custom drivers.
• Data Extraction - Reads records from remote SQL Server databases using JDBC connections and SQL SELECT statements.
• Data Extraction - Extracts data from remote PostgreSQL databases using JDBC connections and SQL select statements.
• Data Partition Parallelism - Supports task-level parallelism by splitting large datasets into independent chunks for concurrent extraction across threads.
• Hub-and-Spoke Data Flows - Utilizes a hub-and-spoke data flow to route information from specialized readers through a central engine to specialized writers.
• Data Component Plugins - Integrates new data sources through the implementation of customizable reader and writer plugins.
• Parallel Data Pipelines - Splits large datasets into concurrent tasks based on primary keys to increase synchronization speed.
• Object Selection Filters - Supports isolating specific objects for extraction through name-based filtering and directory traversal.
• Batch Write Buffering - Groups multiple record writes into a single transaction to increase data ingestion speed and reduce network overhead.
• Custom SQL Execution - Allows the use of custom SQL queries for data extraction to enable complex operations like multi-table joins.
• Export Throughput Limiters - Regulates memory usage and network traffic by capping batch sizes and thread counts during data import.
• Data Synchronization Tools - Provides a tool for bulk data migrations and incremental synchronizations between relational databases and NoSQL stores.
• Data Transformation Functions - Provides built-in functions and custom rules for masking, completing, and filtering data during the migration process.
• Unstructured Text Converters - Translates raw string values from object storage into structured types such as decimals and integers.
• Dirty Data Captures - Provides a dirty-data capture mechanism to intercept and isolate records failing type conversion, ensuring pipeline stability.
• Graph Database Writers - Synchronizes data into Neo4j using Cypher queries to create nodes and relationships.
• Graph Database Exporters - Exports data into graph databases by converting source records into vertices and edges.
• Idempotent Write Strategies - Implements idempotent write strategies that clear target partitions before writing to maintain consistency during failed task retries.
• Bulk Load Optimizations - Moves terabyte-scale data by leveraging temporary distributed storage before triggering optimized bulk load commands.
• Columnar Tabular Storage - Extracts data from optimized columnar storage files including Parquet and ORC for large-scale processing.
• NoSQL Data Writers - Transfers structured data into OTS NoSQL databases with multi-version record support.
• Parallel Storage Writing - Distributes data across multiple threads to write different sub-files simultaneously, improving overall ingestion throughput.
• Relational Database Drivers - Integrates various relational database types by registering JDBC drivers and adding necessary driver files.
• Schema Column Mapping - Selects specific columns for import and rearranges their order to align with the destination schema.
• SQL Data Retrieval - Implements techniques for filtering and extracting specific data from relational tables using SQL WHERE clauses.
• Structured Data File Extractors - Extracts text and field names from structured data files such as CSV and TXT using custom delimiters.
• Tabular Object Storage - Transfers tabular data into object storage using structured formats like Parquet, ORC, and CSV.
• Throughput Controls - Provides mechanisms to cap transfer rates using concurrency channels or byte limits to prevent target system overloading.
• Column Projection - Prunes data during export by selecting a specific subset of columns and defining their output order.
• Throughput Controllers - Regulates memory usage and network traffic by capping concurrency levels and batch sizes during the transfer process.
• Database Batch Writes - Implements batch-based write buffering to group records into single transactions, reducing network overhead and increasing ingestion speed.
• Transfer Retry Mechanisms - Automatically retries failed operations at the thread or task level to ensure completion during network instability.

Star history