# sfu-db/connector-x

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/sfu-db-connector-x).**

2,561 stars · 205 forks · Rust · mit

## Links

- GitHub: https://github.com/sfu-db/connector-x
- Homepage: https://sfu-db.github.io/connector-x
- awesome-repositories: https://awesome-repositories.com/repository/sfu-db-connector-x.md

## Topics

`cpp` `database` `dataframe` `python` `rust` `sql`

## Description

Connector-X is a high-performance SQL data extraction library and bridge for transferring relational database records into memory-efficient data structures. It functions as a parallel database connector and federated query engine capable of executing and joining queries across multiple remote database connections to aggregate data locally.

The project distinguishes itself through a zero-copy approach to data loading, which transfers SQL query results into memory structures without duplicating data. It maximizes throughput by partitioning SQL queries into threads, employing parallel columnar and numerical data downloading to increase ingestion speed.

The system covers broad capabilities in data integration and extensibility, including a pluggable connector interface for custom data source and destination definitions. It provides a type-mapping translation layer to convert source-specific database types into compatible destination formats, specifically supporting high-efficiency ingestion from SQL into Python data frames.

## Tags

### Data & Databases

- [High-Performance Database Extraction](https://awesome-repositories.com/f/data-databases/high-performance-database-extraction.md) — Provides a high-performance method to pull large datasets from relational and analytical databases directly into memory. ([source](https://sfu-db.github.io/connector-x))
- [SQL Data Loaders](https://awesome-repositories.com/f/data-databases/sql-query-interfaces/sql-data-loaders.md) — Provides a high-performance utility for constructing dataframes by extracting data from SQL database queries.
- [SQL to Data Frame Bridges](https://awesome-repositories.com/f/data-databases/sql-to-data-frame-bridges.md) — A bridge for transferring relational database records into memory-efficient data structures for analysis.
- [Query Column Partitioning](https://awesome-repositories.com/f/data-databases/column-based-partitioners/query-column-partitioning.md) — Implements a strategy to maximize network throughput by partitioning SQL queries into parallel column segments.
- [Cross-Warehouse Joins](https://awesome-repositories.com/f/data-databases/data-warehouse-integrations/cross-warehouse-joins.md) — Executes single queries that retrieve and join records across multiple remote database connections. ([source](https://sfu-db.github.io/connector-x))
- [Federated Query Engines](https://awesome-repositories.com/f/data-databases/federated-query-engines.md) — Splits single queries into multiple source requests and aggregates the results locally from disparate databases. ([source](https://github.com/sfu-db/connector-x/blob/main/Federation.md))
- [Parallelized Ingestion](https://awesome-repositories.com/f/data-databases/large-scale-dataset-management/parallelized-ingestion.md) — Distributes data retrieval across multiple concurrent threads by partitioning datasets on specific columns for high-speed ingestion.
- [Zero-Copy Mappings](https://awesome-repositories.com/f/data-databases/object-relational-mapping/zero-copy-mappings.md) — Maps database result sets directly into memory structures without duplication to minimize CPU and memory overhead.
- [SQL Database Connectors](https://awesome-repositories.com/f/data-databases/sql-database-connectors.md) — Provides a parallel database connector that partitions SQL queries into threads to maximize extraction throughput.
- [SQL to Pandas Ingestion](https://awesome-repositories.com/f/data-databases/sql-to-pandas-ingestion.md) — Loads relational database query results into Python data frames with minimal overhead and high efficiency.
- [Python Memory Integration](https://awesome-repositories.com/f/data-databases/zero-copy-data-access/python-memory-integration.md) — Transfers data from relational databases into Python memory structures using a high-performance zero-copy approach. ([source](https://cdn.jsdelivr.net/gh/sfu-db/connector-x@main/README.md))
- [Zero-Copy Data Ingestion](https://awesome-repositories.com/f/data-databases/zero-copy-data-ingestion.md) — Transfers SQL query results into memory-efficient data structures using a zero-copy approach for ingestion. ([source](https://sfu-db.github.io/connector-x/api.html))
- [Custom Connector Development](https://awesome-repositories.com/f/data-databases/data-i-o/custom-connector-development.md) — Supports the development of new data source and destination integrations through custom connection and memory strategies.
- [Custom Data Source Integrations](https://awesome-repositories.com/f/data-databases/data-integration-synchronization/data-integration/custom-data-source-integrations.md) — Provides extensibility to integrate new database connectors via custom connection logic, partitioning strategies, and type parsing. ([source](https://sfu-db.github.io/connector-x/rust-docs/connectorx/))
- [Columnar Partitioning](https://awesome-repositories.com/f/data-databases/data-storage-optimizers/storage-throughput-optimizers/download-throughput-optimizations/columnar-partitioning.md) — Increases throughput by splitting SQL queries into column-based partitions downloaded across separate threads. ([source](https://cdn.jsdelivr.net/gh/sfu-db/connector-x@main/README.md))
- [Data Type Mappings](https://awesome-repositories.com/f/data-databases/data-type-mappings.md) — Converts source-specific database data types into compatible destination types for seamless data transfer. ([source](https://sfu-db.github.io/connector-x/rust-docs/connectorx/))
- [Pluggable Connector Frameworks](https://awesome-repositories.com/f/data-databases/pluggable-storage-drivers/pluggable-connector-frameworks.md) — Provides a standardized interface to integrate heterogeneous database sources and destination formats into the execution engine.
- [Type Mapping Converters](https://awesome-repositories.com/f/data-databases/type-mapping-utilities/type-mapping-converters.md) — Translates source-specific database types into standardized internal formats compatible with destination memory structures.

### DevOps & Infrastructure

- [High-Performance Data Transfer](https://awesome-repositories.com/f/devops-infrastructure/high-performance-data-transfer.md) — Moves large datasets from SQL databases into memory using parallel threads and zero-copy methods for maximum speed.
- [Parallel SQL Loading](https://awesome-repositories.com/f/devops-infrastructure/load-balancing/partitioning-algorithms/parallel-work-partitioning/data-parallelism-frameworks/parallel-sql-loading.md) — Increases data throughput by splitting SQL queries into partitions and downloading them via multiple simultaneous threads.

### Programming Languages & Runtimes

- [Native Execution Engines](https://awesome-repositories.com/f/programming-languages-runtimes/language-interoperability/native-c-interoperability/native-execution-engines.md) — Uses a native execution engine written in Rust to ensure high-performance data movement and efficient memory management.