2 repository-uri
Awesome GitHub RepositoriesParallel Data Pipelines
Spawns multiple concurrent threads that each read a disjoint range of source data, transform it, and feed it into a dedicated COPY stream.
Distinct from Parallel Task Spawning: Distinct from Parallel Task Spawning: focuses on data pipeline parallelism with disjoint ranges and COPY streams, not general task spawning.
Explore 2 awesome GitHub repositories matching software engineering & architecture · Parallel Data Pipelines. Refine with filters or upvote what's useful.
Awesome Parallel Data Pipelines GitHub Repositories
Găsește cele mai bune repo-uri cu AI.Vom căuta cele mai potrivite repository-uri folosind AI.
- alibaba/datax
alibaba/DataX
17,241Vezi pe GitHubDataX is a distributed data integration framework and plugin-based ETL tool designed for synchronizing large datasets between heterogeneous sources and destinations. It functions as a JDBC data migration engine and offline synchronization tool, enabling the movement of data between relational databases, NoSQL stores, and object storage. The system utilizes a plugin-based connector architecture that decouples reader and writer logic, allowing it to map and transform data types across different storage engines using a standardized internal representation. This design supports heterogeneous data
Splits large datasets into concurrent tasks based on primary keys to increase synchronization speed.
Java - dimitri/pgloader
dimitri/pgloader
6,295Vezi pe GitHubpgloader is a command-line tool that automates the migration of data and schema from various source databases and file formats into PostgreSQL. It combines schema discovery, parallel data pipelines, and type casting into a single, declarative workflow, using PostgreSQL's COPY protocol for high-throughput bulk loading. The tool distinguishes itself by compiling a dedicated command language into concurrent reader-writer pipelines that handle schema introspection, data transformation, and error-resilient batch processing. It supports migrating entire databases from MySQL, MS SQL, SQLite, and Pos
Spawns multiple concurrent threads that each read a disjoint range of source data, transform it, and feed it into a dedicated COPY stream.
Common Lispclozure-clcommon-lispcsv
