Awesome GitHub RepositoriesParallelized Ingestion

LanceDB is a vector database and columnar data store designed to function as a versioned dataset manager and vector search engine. It serves as a high-performance backend for indexing and retrieving high-dimensional embeddings, providing the foundation for machine learning data pipelines. The system distinguishes itself through a combination of cloud-native object storage and immutable version tracking, allowing for data time-travel and reproducible AI experiments. It integrates hybrid search capabilities, merging dense vector similarity with BM25 full-text search and SQL-like scalar filters

Hazelcast is a distributed data platform that combines an in-memory data grid with a stream processing engine to support real-time analytics and event-driven applications. It functions as a partitioned, distributed key-value store that replicates data across cluster nodes to provide low-latency access and high availability. The platform also serves as a distributed SQL query engine, allowing users to execute standard SQL statements against both in-memory datasets and external data sources. What distinguishes Hazelcast is its use of a distributed consensus subsystem to maintain strongly consis

pgdog is a PostgreSQL sharding proxy, distributed SQL router, and connection pooler. It is designed to enable horizontal data distribution by splitting tables and indices across multiple independent servers to scale storage and processing capacity. The project distinguishes itself through online resharding capabilities, using logical replication to move data between shards without application downtime. It supports multiple routing strategies, including hash, list, and range-based query routing, and manages distributed atomic transactions using a two-phase commit process to ensure consistency

NeMo-Retriever is a framework designed for building end-to-end document ingestion and retrieval-augmented generation pipelines. It provides a suite of tools for processing, classifying, and structuring diverse file formats, transforming raw enterprise data into searchable information assets for generative artificial intelligence applications. The system distinguishes itself through its specialized capabilities for parsing complex document layouts, including tables, charts, and infographics, using integrated optical character recognition and multi-modal extraction. It utilizes a microservice-b

Connector-X is a high-performance SQL data extraction library and bridge for transferring relational database records into memory-efficient data structures. It functions as a parallel database connector and federated query engine capable of executing and joining queries across multiple remote database connections to aggregate data locally. The project distinguishes itself through a zero-copy approach to data loading, which transfers SQL query results into memory structures without duplicating data. It maximizes throughput by partitioning SQL queries into threads, employing parallel columnar a

هذا المشروع عبارة عن إضافة قاعدة بيانات تتيح التوسع الأفقي وتقسيم البيانات الموزعة عبر عقد PostgreSQL متعددة. يعمل كحل برمجيات وسيطة يوزع جداول قاعدة البيانات الكبيرة عبر عنقود لزيادة سعة التخزين الإجمالية وتحسين أداء الكتابة لمجموعات البيانات المتنامية. يستخدم النظام التقسيم القائم على التجزئة (hash-based partitioning) لتوزيع سجلات الجدول عبر العقد المادية ويستخدم كتالوج بيانات وصفية مركزياً لتتبع مواقع الأجزاء (shards). يوفر توجيهاً شفافاً للاستعلام، والذي يوجه أوامر SQL القياسية من عقدة منسقة إلى عقد العمل المناسبة دون الحاجة إلى تعديلات على منطق التطبيق الحالي. يتضمن إطار العمل أدوات للحفاظ على تكرار البيانات وسلامة العنقود من خلال تكرار الأجزاء وعمليات إصلاح الاتساق الآلية. كما يدعم استيعاب البيانات عالي الإنتاجية، باستخدام المعالجة المتزامنة لاستيراد مجموعات بيانات ضخمة إلى جداول موزعة.