2 Repos
Awesome GitHub RepositoriesParallel Data Pipelines
Spawns multiple concurrent threads that each read a disjoint range of source data, transform it, and feed it into a dedicated COPY stream.
Distinct from Parallel Task Spawning: Distinct from Parallel Task Spawning: focuses on data pipeline parallelism with disjoint ranges and COPY streams, not general task spawning.
Explore 2 awesome GitHub repositories matching software engineering & architecture · Parallel Data Pipelines. Refine with filters or upvote what's useful.
Awesome Parallel Data Pipelines GitHub Repositories
Finde die besten Repos mit KI.Wir suchen mit KI nach den am besten passenden Repositories.
- alibaba/datax
alibaba/DataX
17,241Auf GitHub ansehenDataX is a distributed data integration framework and plugin-based ETL tool designed for synchronizing large datasets between heterogeneous sources and destinations. It functions as a JDBC data migration engine and offline synchronization tool, enabling the movement of data between relational databases, NoSQL stores, and object storage. The system utilizes a plugin-based connector architecture that decouples reader and writer logic, allowing it to map and transform data types across different storage engines using a standardized internal representation. This design supports heterogeneous data
Splits large datasets into concurrent tasks based on primary keys to increase synchronization speed.
Java - dimitri/pgloader
dimitri/pgloader
6,295Auf GitHub ansehenpgloader is a command-line tool that automates the migration of data and schema from various source databases and file formats into PostgreSQL. It combines schema discovery, parallel data pipelines, and type casting into a single, declarative workflow, using PostgreSQL's COPY protocol for high-throughput bulk loading. The tool distinguishes itself by compiling a dedicated command language into concurrent reader-writer pipelines that handle schema introspection, data transformation, and error-resilient batch processing. It supports migrating entire databases from MySQL, MS SQL, SQLite, and Pos
Spawns multiple concurrent threads that each read a disjoint range of source data, transform it, and feed it into a dedicated COPY stream.
Common Lispclozure-clcommon-lispcsv
