Awesome GitHub RepositoriesStreaming Source and Sink Integration

This project is a collection of educational resources and reference implementations for the Apache Flink stream processing framework. It provides a learning resource focused on mastering distributed stream processing through implementation guides, performance tuning tutorials, and practical examples. The repository features detailed walkthroughs for building real-time data pipelines using the DataStream and Table APIs. It includes specific integration examples for connecting Apache Flink with Kafka brokers and Elasticsearch indices, as well as reference implementations for real-time deduplica

Nebula is a distributed graph database designed for storing and querying massive volumes of interconnected vertices and edges across a horizontally scalable cluster. It functions as a Kubernetes-native database and a distributed graph analytics engine, utilizing a Raft-based distributed store to ensure strong consistency and high availability. The system features an OpenCypher query engine for performing complex graph traversals and pattern matching. It distinguishes itself with a decoupled compute-storage architecture and a shared-nothing distributed design, allowing query processing and dat

This project is a streaming data integration framework that captures real-time database changes and synchronizes them with downstream systems. It operates as a distributed streaming ETL and database synchronizer, reading database logs and snapshots to propagate row-level modifications to target sinks. The system supports declarative data integration, allowing users to define source-to-sink data flows using SQL or YAML configurations. It distinguishes itself by automating schema evolution to maintain synchronization when source structures change and ensuring exactly-once delivery and processin

lakeFS is a data lake versioning system that provides Git-like branching and commits for large datasets stored in object storage. It functions as a version control layer, enabling the creation of immutable snapshots, atomic commits, and zero-copy branching to create isolated environments for data experimentation without duplicating physical files. The system serves as an S3-compatible storage gateway and an Iceberg REST catalog, allowing standard cloud storage protocols and compatible clients to manage versioned tables. It acts as a data quality gatekeeper by using an event-driven hook system

oboe.js este un parser JSON de streaming și un client HTTP stream conceput pentru procesarea incrementală a datelor. Permite aplicațiilor să acționeze asupra obiectelor JSON individuale pe măsură ce sosesc dintr-o cerere de rețea sau un stream, înainte ca descărcarea completă să fie finalizată. Biblioteca funcționează ca un transformator de stream JSON, permițând modificarea, filtrarea și maparea nodurilor în timp real. Dispune de selecție de noduri bazată pe path pentru a declanșa callback-uri atunci când sunt identificate tipare specifice și poate închide conexiunea de rețea imediat ce datele necesare sunt găsite, pentru a economisi lățimea de bandă. Pentru a gestiona seturi de date mai mari decât memoria disponibilă a sistemului, biblioteca utilizează gestionarea memoriei bazată pe stream pentru a elimina nodurile procesate. De asemenea, suportă fuziunea înregistrărilor istorice cu actualizări în timp real prin conexiuni de lungă durată și oferă posibilitatea de a recupera date parțiale dacă o conexiune este întreruptă.

Kotlinx-io is a multiplatform library designed for input and output operations, providing a unified interface for streaming data, managing byte buffers, and interacting with local filesystems. It serves as a cross-platform abstraction layer that standardizes how applications handle data movement across different operating systems and hardware architectures. The library distinguishes itself by providing high-performance tools for both mutable and immutable byte sequences. It utilizes segmented memory pools and direct memory access to minimize allocation overhead and prevent unnecessary data co

Data warehouse learning is a reference implementation of a real-time stream processing system and open-source data lakehouse architecture. It combines stream processing engines, open lakehouse formats, and analytical data warehouses into a complete e-commerce data warehouse system built for both offline and real-time analytics pipelines. The project implements hybrid data warehouse architectures utilizing multi-layer storage models and stream-batch processing pipelines. It features change data capture pipelines that stream database transaction logs into messaging systems, progressive data tra