sfu-db/connector-x

Features

• High-Performance Database Extraction - Provides a high-performance method to pull large datasets from relational and analytical databases directly into memory.
• SQL Data Loaders - Provides a high-performance utility for constructing dataframes by extracting data from SQL database queries.
• SQL to Data Frame Bridges - A bridge for transferring relational database records into memory-efficient data structures for analysis.
• Query Column Partitioning - Implements a strategy to maximize network throughput by partitioning SQL queries into parallel column segments.
• Cross-Warehouse Joins - Executes single queries that retrieve and join records across multiple remote database connections.
• Federated Query Engines - Splits single queries into multiple source requests and aggregates the results locally from disparate databases.
• Parallelized Ingestion - Distributes data retrieval across multiple concurrent threads by partitioning datasets on specific columns for high-speed ingestion.
• Zero-Copy Mappings - Maps database result sets directly into memory structures without duplication to minimize CPU and memory overhead.
• SQL Database Connectors - Provides a parallel database connector that partitions SQL queries into threads to maximize extraction throughput.
• SQL to Pandas Ingestion - Loads relational database query results into Python data frames with minimal overhead and high efficiency.
• Python Memory Integration - Transfers data from relational databases into Python memory structures using a high-performance zero-copy approach.
• Zero-Copy Data Ingestion - Transfers SQL query results into memory-efficient data structures using a zero-copy approach for ingestion.
• High-Performance Data Transfer - Moves large datasets from SQL databases into memory using parallel threads and zero-copy methods for maximum speed.
• Parallel SQL Loading - Increases data throughput by splitting SQL queries into partitions and downloading them via multiple simultaneous threads.
• Native Execution Engines - Uses a native execution engine written in Rust to ensure high-performance data movement and efficient memory management.
• Custom Connector Development - Supports the development of new data source and destination integrations through custom connection and memory strategies.
• Custom Data Source Integrations - Provides extensibility to integrate new database connectors via custom connection logic, partitioning strategies, and type parsing.
• Columnar Partitioning - Increases throughput by splitting SQL queries into column-based partitions downloaded across separate threads.
• Data Type Mappings - Converts source-specific database data types into compatible destination types for seamless data transfer.
• Pluggable Connector Frameworks - Provides a standardized interface to integrate heterogeneous database sources and destination formats into the execution engine.
• Type Mapping Converters - Translates source-specific database types into standardized internal formats compatible with destination memory structures.

Star history