Awesome GitHub RepositoriesChange Data Capture Streams

RethinkDB is a distributed, document-oriented database designed to store and manage JSON-formatted data across scalable clusters. It utilizes a custom log-structured storage engine with B-Tree indexing to ensure high-performance disk I/O and data persistence. The system maintains high availability through automatic sharding and replication, employing a primary-replica voting consensus mechanism to handle node failures and ensure consistent cluster operations. A defining characteristic of the platform is its reactive changefeed engine, which allows applications to subscribe to live data update

This project is a reactive, offline-first NoSQL database engine designed for JavaScript applications. It provides a robust framework for managing application state by synchronizing data across browsers, mobile devices, and server-side runtimes. By treating local storage as the primary source of truth, it enables applications to remain functional without network connectivity, automatically reconciling changes with remote backends once a connection is restored. The database distinguishes itself through a modular architecture that supports cross-environment synchronization and high-performance d

rqlite is a distributed relational database that replicates SQLite data across a cluster using the Raft consensus algorithm. It functions as a fault-tolerant storage system that provides high availability and a web API for executing SQL queries and managing relational data without requiring native database drivers. The system distinguishes itself by using an HTTP SQL interface to expose database operations and cluster management. It features a real-time change data capture stream that pushes database mutations to external HTTP endpoints via webhooks and supports the scaling of read throughput

The AWS Cloud Development Kit is an infrastructure-as-code framework that enables developers to define and provision cloud resources using familiar programming languages. By utilizing construct-based synthesis, it translates high-level, object-oriented code into declarative templates, allowing for the automated management of complex cloud environments through a centralized, code-driven control plane. The framework distinguishes itself through its ability to model infrastructure as a dependency-aware resource graph, ensuring that components are provisioned and updated in the correct order. It

DataHub is a metadata management platform designed to unify technical, operational, and business context across diverse data ecosystems. By utilizing a graph-based metadata model and an event-driven ingestion architecture, it creates a centralized source of truth that maps complex data relationships, lineage, and ownership. This foundational framework enables organizations to maintain a synchronized view of their data landscape, supporting both human-led discovery and automated data operations. The platform distinguishes itself through its focus on grounding artificial intelligence and autono

Nango is an open-source platform that connects applications to external APIs by managing authentication, data synchronization, and custom function execution. It provides a managed runtime for TypeScript integration functions, handling OAuth flows, credential storage, and token refresh for hundreds of external APIs while keeping secrets isolated from application code. The platform distinguishes itself by exposing integration functions as discoverable tools for AI agents through an MCP server or API, with per-user credential isolation that keeps provider secrets out of the agent loop. It offers

Electric is a Postgres data synchronization engine and replication proxy designed to enable local-first software. It replicates data from Postgres databases to client-side stores in real time using logical replication, allowing applications to maintain a local embedded database for offline access and low-latency updates. The system distinguishes itself by using shapes to filter and authorize specific subsets of database rows and columns before streaming them to clients or edge workers. It further supports multi-user collaboration by integrating a conflict-free replicated data type framework t

Boto3 is the AWS SDK for Python, providing a programmatic interface for managing and automating AWS cloud infrastructure and services. It serves as a cloud management API client and resource manager for provisioning, configuring, and scaling virtual servers, databases, and storage. The library enables the implementation of infrastructure-as-code through declarative templates and scripts, allowing for the deployment of identical resource stacks across multiple accounts and geographic regions. It also provides a framework for coordinating distributed workflows, serverless functions, and contain

OpenMediaVault is a Debian-based network attached storage (NAS) operating system that turns a computer into a dedicated storage appliance, managed entirely through a browser-based control panel. It provides a unified configuration interface for managing file sharing protocols, RAID arrays, user permissions, and network services, with extensibility through a modular plugin system that adds new capabilities without modifying the core installation. The platform supports setup and management of multiple RAID levels for data redundancy or performance, and exposes file storage over protocols includ

Hazelcast is a distributed data platform that combines an in-memory data grid with a stream processing engine to support real-time analytics and event-driven applications. It functions as a partitioned, distributed key-value store that replicates data across cluster nodes to provide low-latency access and high availability. The platform also serves as a distributed SQL query engine, allowing users to execute standard SQL statements against both in-memory datasets and external data sources. What distinguishes Hazelcast is its use of a distributed consensus subsystem to maintain strongly consis

Ibis is a portable Python dataframe library and multi-backend query engine that provides a unified interface for executing data transformations across diverse compute engines. It functions as a Python SQL expression compiler and dialect transpiler, allowing users to define data logic once and execute it across cloud warehouses, embedded databases, and distributed clusters without rewriting code. The project distinguishes itself through a database backend abstraction that decouples transformation logic from the underlying execution engine. It enables polyglot data workflows by mixing raw SQL s

Airweave is a unified AI knowledge base platform that syncs data from external APIs into a searchable layer for retrieval-augmented generation. It provides a pre-built data connector library and a framework for building custom connectors, enabling the extraction, transformation, and synchronization of structured and unstructured data from SaaS applications. The platform includes a hybrid vector retrieval system that combines semantic, neural, and keyword search strategies to deliver grounded context for AI agents. The platform distinguishes itself through an agentic search engine that iterati

DataX Web is a web-based management platform for scheduling, building, executing, and monitoring distributed data synchronization jobs powered by DataX. It provides a visual console for creating and managing DataX tasks without manual JSON configuration, with a distributed executor cluster that auto-registers worker nodes and supports configurable routing and blocking strategies for task distribution. The platform offers cron-based task scheduling with dynamic start, stop, and immediate status changes, along with incremental sync capabilities that pass dynamic parameters to extract only new o

Arroyo is a high-performance stream processing platform built in Rust. It executes continuous SQL queries on streaming data with event-time semantics, enabling accurate windowed aggregations, joins, and stateful computations on unbounded event streams. The platform uses native Rust execution for high throughput and low latency, with periodic checkpointing for exactly-once fault tolerance and horizontal scaling across distributed workers. The system integrates deeply with Kafka for reading and writing topics with exactly-once delivery and supports change data capture (CDC) from MySQL and Postg

YDB ist eine verteilte SQL-Datenbank und analytische Engine, die auf horizontale Skalierbarkeit und starke Konsistenz ausgelegt ist. Sie fungiert als Multi-Model-System, das transaktionale und analytische Workloads durch eine verteilte Architektur unterstützt, die serialisierbare ACID-Transaktionen bietet. Das System zeichnet sich durch seine breite Protokollkompatibilität aus und implementiert das PostgreSQL-Wire-Protokoll für Standard-SQL-Treiber sowie das Kafka-Protokoll für Messaging und Streaming. Es dient zudem als Vektordatenbank und unterstützt Vektorindizes sowie die Suche nach ungefähren nächsten Nachbarn (Approximate Nearest Neighbor) für semantische Suche und Embeddings. Die Plattform verwaltet Daten unter Verwendung eines hybriden Speichermodells mit zeilen- und spaltenorientierten Formaten und nutzt vektorisierte Abfrageausführung für Analysen im Petabyte-Maßstab. Ihr operativer Bereich umfasst Change-Data-Capture-Streaming, persistente Queues mit Exactly-Once-Garantie und Multi-Zone-Hochverfügbarkeit. Deployment und Lebenszyklusmanagement werden durch einen Kubernetes-Operator und Infrastructure-as-Code-Provisioning unterstützt.

Dieses Projekt ist ein Markdown-Static-Site-Generator und Dokumentationsserver, der Dateien und Ordnerstrukturen in responsive Websites umwandelt. Es fungiert als Confluence-Dokumentations-Publisher und ermöglicht das direkte Hochladen und Synchronisieren von verarbeiteten Markdown-Inhalten auf Enterprise-Knowledge-Base-Server. Das System zeichnet sich durch seine technischen Rendering-Fähigkeiten aus, indem es Mermaid-Syntax in visuelle Diagramme und TeX-Ausdrücke in mathematische Formeln umwandelt. Es unterstützt mehrsprachige Dokumentation durch die Erstellung lokalisierter Versionen einer Website mit dedizierten Navigationsstrukturen. Die Plattform bietet eine vollständige Content-Pipeline, die hierarchische Navigationsmenüs, Code-Syntax-Highlighting und ein System zur Verwaltung der Seitenreihenfolge über Dateinamen-Präfixe umfasst. Zudem verfügt sie über ein flexibles Erweiterungs-Framework für benutzerdefinierte Content-Generatoren und die programmatische Manipulation von Seitenbäumen. Die Umgebung kann als portabler Docker-Container bereitgestellt werden und enthält einen lokalen Preview-Server zum Testen der Dokumentation vor der finalen Generierung.

Dieses Projekt ist ein Change-Data-Capture-System und eine Synchronisationsschicht, die Daten aus MySQL-Datenbanken in Elasticsearch-Indizes verschiebt. Es fungiert als Relational-to-Document-Mapper, der Datenbanktabellen in durchsuchbare Dokumente umwandelt, um Echtzeit-Datenintegration und Volltextsuche zu ermöglichen. Der Synchronizer zeichnet sich durch die Unterstützung der Denormalisierung relationaler Daten aus, die Eins-zu-Viele-Datenbank-Joins in Eltern-Kind-Dokumentstrukturen umwandelt. Er ermöglicht zudem die Aggregation partitionierter Tabellen unter Verwendung regulärer Ausdrücke, um mehrere Datenbanktabellen in einem einzigen Suchindex zu gruppieren. Das System deckt umfassende Datenmapping- und Transformationsfunktionen ab, einschließlich Feldtypkonvertierung, Schema-Mapping und synchronisierter Filterung von Feldern. Es verwendet ein Pipeline-basiertes Verarbeitungsmodell zum Dekodieren und Zusammenführen von Feldern und nutzt sowohl Snapshot-basiertes Initial-Loading für Baselines als auch Binary-Log-Streaming für Echtzeit-Updates.

RavenDB is a multi-model NoSQL document database designed for high-performance, ACID-compliant data storage. It persists structured information as schema-flexible JSON documents and utilizes a unit-of-work session pattern to track entity changes and batch modifications into atomic transactions. The platform is built on a distributed architecture that supports horizontal scaling through sharding and ensures high availability via multi-node, master-to-master cluster replication. The database distinguishes itself through a self-optimizing query engine that automatically creates and maintains ind

This project is a collection of structured study notes and conceptual breakdowns designed for the AWS Certified Cloud Practitioner exam. It serves as a technical reference and study guide, organizing cloud service details and architectural principles to assist in certification preparation. The knowledge base is built using markdown files and includes curated cheat sheets and interactive mind-map visualizations. These tools map complex certification topics into visual hierarchies to enable drill-down study paths and rapid revision. The materials cover a wide range of cloud capabilities, inclu

The platform is a distributed system designed for real-time data monitoring, continuous graph-based query processing, and reactive event automation. It functions as a middleware solution that tracks state changes in external databases and systems, evaluating these streams against graph patterns to identify significant events and state transitions without the need for manual polling. The platform distinguishes itself through its ability to synchronize state updates across distributed environments, including real-time updates to vector databases for AI applications. It utilizes a pluggable conn