6 repository-uri
Awesome GitHub RepositoriesParallelized Ingestion
Writing massive datasets using parallelized batches to prevent memory exhaustion.
Distinct from Large-Scale Dataset Management: Specializes large-scale management by focusing on the parallel writing process of the ingestion phase.
Explore 6 awesome GitHub repositories matching data & databases · Parallelized Ingestion. Refine with filters or upvote what's useful.
Awesome Parallelized Ingestion GitHub Repositories
Găsește cele mai bune repo-uri cu AI.Vom căuta cele mai potrivite repository-uri folosind AI.
- lancedb/lancedb
lancedb/lancedb
9,031Vezi pe GitHubLanceDB is a vector database and columnar data store designed to function as a versioned dataset manager and vector search engine. It serves as a high-performance backend for indexing and retrieving high-dimensional embeddings, providing the foundation for machine learning data pipelines. The system distinguishes itself through a combination of cloud-native object storage and immutable version tracking, allowing for data time-travel and reproducible AI experiments. It integrates hybrid search capabilities, merging dense vector similarity with BM25 full-text search and SQL-like scalar filters
Writes massive datasets from files or iterators using parallelized batches to avoid memory exhaustion.
HTMLapproximate-nearest-neighbor-searchimage-searchnearest-neighbor-search - hazelcast/hazelcast
hazelcast/hazelcast
6,570Vezi pe GitHubHazelcast is a distributed data platform that combines an in-memory data grid with a stream processing engine to support real-time analytics and event-driven applications. It functions as a partitioned, distributed key-value store that replicates data across cluster nodes to provide low-latency access and high availability. The platform also serves as a distributed SQL query engine, allowing users to execute standard SQL statements against both in-memory datasets and external data sources. What distinguishes Hazelcast is its use of a distributed consensus subsystem to maintain strongly consis
Distributes connector tasks across cluster nodes to parallelize ingestion and handle configuration changes.
Javabig-datacachingdata-in-motion - pgdogdev/pgdog
pgdogdev/pgdog
3,361Vezi pe GitHubpgdog is a PostgreSQL sharding proxy, distributed SQL router, and connection pooler. It is designed to enable horizontal data distribution by splitting tables and indices across multiple independent servers to scale storage and processing capacity. The project distinguishes itself through online resharding capabilities, using logical replication to move data between shards without application downtime. It supports multiple routing strategies, including hash, list, and range-based query routing, and manages distributed atomic transactions using a two-phase commit process to ensure consistency
Parses data streams and distributes rows to multiple shards in parallel to accelerate bulk loading.
Rustload-balancerpoolerpostgresql - nvidia/nemo-retriever
NVIDIA/NeMo-Retriever
2,940Vezi pe GitHubNeMo-Retriever is a framework designed for building end-to-end document ingestion and retrieval-augmented generation pipelines. It provides a suite of tools for processing, classifying, and structuring diverse file formats, transforming raw enterprise data into searchable information assets for generative artificial intelligence applications. The system distinguishes itself through its specialized capabilities for parsing complex document layouts, including tables, charts, and infographics, using integrated optical character recognition and multi-modal extraction. It utilizes a microservice-b
Executes concurrent document splitting and chunking workflows to maximize throughput for large-scale enterprise datasets.
Python - sfu-db/connector-x
sfu-db/connector-x
2,561Vezi pe GitHubConnector-X is a high-performance SQL data extraction library and bridge for transferring relational database records into memory-efficient data structures. It functions as a parallel database connector and federated query engine capable of executing and joining queries across multiple remote database connections to aggregate data locally. The project distinguishes itself through a zero-copy approach to data loading, which transfers SQL query results into memory structures without duplicating data. It maximizes throughput by partitioning SQL queries into threads, employing parallel columnar a
Distributes data retrieval across multiple concurrent threads by partitioning datasets on specific columns for high-speed ingestion.
Rustcppdatabasedataframe - citusdata/pg_shard
citusdata/pg_shard
1,062Vezi pe GitHubAcest proiect este o extensie de bază de date care permite scalarea orizontală și partiționarea distribuită a datelor pe mai multe noduri PostgreSQL. Funcționează ca o soluție middleware care distribuie tabelele mari de baze de date într-un cluster pentru a crește capacitatea totală de stocare și a îmbunătăți performanța de scriere pentru seturile de date în creștere. Sistemul folosește partiționarea bazată pe hash pentru a distribui înregistrările tabelelor pe noduri fizice și utilizează un catalog central de metadate pentru a urmări locațiile shard-urilor. Oferă rutare transparentă a interogărilor, care direcționează comenzile SQL standard de la un nod coordonator către nodurile lucrătoare (worker nodes) corespunzătoare, fără a necesita modificări ale logicii aplicației existente. Framework-ul include instrumente pentru menținerea redundanței datelor și a integrității clusterului prin replicarea shard-urilor și procese automatizate de reparare a consistenței. De asemenea, suportă ingestia de date cu throughput ridicat, utilizând procesarea concurentă pentru a importa seturi de date masive în tabele distribuite.
Loads massive volumes of information into a database cluster using concurrent processes to maximize throughput and reduce import times.
C
