2 Repos
Awesome GitHub RepositoriesParallel Storage Writing
Distributing data writes across multiple threads to write different sub-files or partitions simultaneously.
Distinct from Parallel Shard Reads and Writes: Focuses on improving overall throughput via multi-threaded file writing, unlike shard-level lock contention avoidance.
Explore 2 awesome GitHub repositories matching data & databases · Parallel Storage Writing. Refine with filters or upvote what's useful.
Awesome Parallel Storage Writing GitHub Repositories
Finde die besten Repos mit KI.Wir suchen mit KI nach den am besten passenden Repositories.
- alibaba/datax
alibaba/DataX
17,241Auf GitHub ansehenDataX is a distributed data integration framework and plugin-based ETL tool designed for synchronizing large datasets between heterogeneous sources and destinations. It functions as a JDBC data migration engine and offline synchronization tool, enabling the movement of data between relational databases, NoSQL stores, and object storage. The system utilizes a plugin-based connector architecture that decouples reader and writer logic, allowing it to map and transform data types across different storage engines using a standardized internal representation. This design supports heterogeneous data
Distributes data across multiple threads to write different sub-files simultaneously, improving overall ingestion throughput.
Java - eto-ai/lance
eto-ai/lance
6,671Auf GitHub ansehenLance is a versioned columnar data format and storage engine designed as a multimodal AI lakehouse. It serves as a vector database storage engine and a cloud object store dataset manager, organizing images, video, audio, and embeddings into a unified format optimized for machine learning workflows. The project distinguishes itself by combining a columnar layout for structured data with a specialized blob store for large multimodal tensors. It implements a hybrid search engine that integrates vector similarity search, full-text search, and SQL analytics on a single dataset, supported by a stor
Generates data fragments in parallel across multiple nodes and commits them into a single dataset.
Rust
