2 repository-uri
Awesome GitHub RepositoriesParallel Storage Writing
Distributing data writes across multiple threads to write different sub-files or partitions simultaneously.
Distinct from Parallel Shard Reads and Writes: Focuses on improving overall throughput via multi-threaded file writing, unlike shard-level lock contention avoidance.
Explore 2 awesome GitHub repositories matching data & databases · Parallel Storage Writing. Refine with filters or upvote what's useful.
Awesome Parallel Storage Writing GitHub Repositories
Găsește cele mai bune repo-uri cu AI.Vom căuta cele mai potrivite repository-uri folosind AI.
- alibaba/datax
alibaba/DataX
17,241Vezi pe GitHubDataX is a distributed data integration framework and plugin-based ETL tool designed for synchronizing large datasets between heterogeneous sources and destinations. It functions as a JDBC data migration engine and offline synchronization tool, enabling the movement of data between relational databases, NoSQL stores, and object storage. The system utilizes a plugin-based connector architecture that decouples reader and writer logic, allowing it to map and transform data types across different storage engines using a standardized internal representation. This design supports heterogeneous data
Distributes data across multiple threads to write different sub-files simultaneously, improving overall ingestion throughput.
Java - eto-ai/lance
eto-ai/lance
6,671Vezi pe GitHubLance is a versioned columnar data format and storage engine designed as a multimodal AI lakehouse. It serves as a vector database storage engine and a cloud object store dataset manager, organizing images, video, audio, and embeddings into a unified format optimized for machine learning workflows. The project distinguishes itself by combining a columnar layout for structured data with a specialized blob store for large multimodal tensors. It implements a hybrid search engine that integrates vector similarity search, full-text search, and SQL analytics on a single dataset, supported by a stor
Generates data fragments in parallel across multiple nodes and commits them into a single dataset.
Rust
