Streaming, queues and change data capture

This project provides an integrated backend platform built around a relational database. It automatically generates REST and GraphQL APIs from database schemas, allowing for direct data interaction through standard requests and client libraries. The platform includes a comprehensive authentication system that manages user identity, session handling, and fine-grained access control through database-native row-level security policies. Beyond core data management, the platform offers specialized services for object storage, vector data processing for semantic search, and real-time communication features like broadcast messaging and database change subscriptions. It also supports server-side logic execution through globally distributed edge functions, database-resident functions, and a native job scheduler for automated tasks. Developers can manage the entire project lifecycle using a command-line interface and containerized local development environments. The platform supports both managed cloud services and self-hosted deployments, providing options for infrastructure control and data sovereignty.

Canal is a database replication middleware that performs change data capture by simulating a database replica. It monitors transaction logs to stream incremental data modifications to downstream systems in real time, acting as an event streaming infrastructure that transforms low-level binary logs into structured, consumable message streams. The project distinguishes itself through a high-throughput architecture that utilizes concurrent multi-threaded parsing and stateful log position tracking to ensure reliable data delivery. It employs a pluggable sink architecture that decouples data extraction from destination storage, allowing for flexible routing to various message queues or secondary databases. Users can manage data consistency and throughput through configurable message ordering and batching strategies, while dynamic configuration injection enables runtime adjustments to routing rules without requiring service restarts. The platform includes comprehensive operational tools for monitoring system health and performance, including metrics for transaction latency and network bandwidth. It supports secure network connectivity for data transmission and provides specialized integration for cloud-based environments, including the ability to retrieve archived logs from object storage. The service is designed for containerized deployment, incorporating automated resource management to maintain synchronization pipelines.

RethinkDB is a distributed, document-oriented database designed to store and manage JSON-formatted data across scalable clusters. It utilizes a custom log-structured storage engine with B-Tree indexing to ensure high-performance disk I/O and data persistence. The system maintains high availability through automatic sharding and replication, employing a primary-replica voting consensus mechanism to handle node failures and ensure consistent cluster operations. A defining characteristic of the platform is its reactive changefeed engine, which allows applications to subscribe to live data updates. Instead of polling for changes, developers can maintain persistent cursors on tables to stream document modifications in real-time. This is complemented by a fluent, functional query language that translates native code constructs into optimized, parallelized execution plans. By embedding these queries directly into application code, the system provides a type-safe interface that helps prevent injection vulnerabilities while enabling complex data manipulation and aggregation. The platform provides a comprehensive suite of administrative tools for managing production environments, including granular user permissions, TLS network encryption, and visual cluster monitoring. It supports advanced data modeling through document embedding and cross-table linking, as well as specialized geospatial processing for proximity-based queries. The system is designed for integration with modern web frameworks and message brokers, facilitating real-time synchronization with external services and search engines. RethinkDB is configured via key-value files and command-line interfaces, with support for containerized deployment and automated infrastructure orchestration.

TiDB is a horizontally scalable, distributed SQL database designed to provide consistent transactional storage and high-performance analytical processing within a single unified architecture. It utilizes a decoupled compute-storage design and a distributed key-value storage layer to ensure horizontal scalability and efficient range-based queries. By employing a consensus-based replication algorithm, the system maintains high availability and automatic failover across multiple nodes and geographical regions. The platform distinguishes itself through its hybrid transactional and analytical processing capabilities, which allow complex SQL queries to run against replicated columnar data without disrupting primary transactional workloads. It also integrates high-dimensional vector search functionality, enabling semantic similarity queries directly alongside traditional relational data. To support diverse operational needs, the system provides native tools for real-time data streaming, seamless migration from external database systems, and multi-region disaster recovery. The database is built for cloud-native environments, offering comprehensive lifecycle management through Kubernetes operators that automate deployment, scaling, and rolling upgrades. It maintains compatibility with standard SQL interfaces, allowing applications to connect using common drivers while managing complex concurrency through pessimistic transaction handling. Detailed documentation and command-line utilities are available to assist with cluster orchestration, performance troubleshooting, and the configuration of production-grade topologies.

The AWS Cloud Development Kit is an infrastructure-as-code framework that enables developers to define and provision cloud resources using familiar programming languages. By utilizing construct-based synthesis, it translates high-level, object-oriented code into declarative templates, allowing for the automated management of complex cloud environments through a centralized, code-driven control plane. The framework distinguishes itself through its ability to model infrastructure as a dependency-aware resource graph, ensuring that components are provisioned and updated in the correct order. It employs a language-agnostic intermediate representation to synthesize these definitions into platform-specific configurations, while supporting aspect-oriented policy injection to apply security and compliance rules across infrastructure definitions during the synthesis phase. Beyond core provisioning, the project provides a modular component registry for distributing and reusing pre-configured infrastructure building blocks. It supports multi-account orchestration, allowing for the deployment of consistent resource sets across different regions and accounts from a single template, and includes capabilities for detecting infrastructure drift to ensure deployed environments remain aligned with their defined state. The project is distributed as a software development kit, providing programmatic interfaces to manage the full lifecycle of cloud resources and integrate infrastructure definitions directly into application codebases.

Kafka is a distributed event streaming platform designed for capturing, storing, and processing real-time data streams across interconnected nodes. It functions as a distributed commit log, providing a fault-tolerant storage mechanism that records state changes sequentially to ensure data consistency and durability across distributed environments. The platform distinguishes itself through a partitioned commit log architecture that enables horizontal scaling and parallel processing of data streams. It integrates a stream processing engine for continuous transformations and aggregations, while utilizing log-structured, append-only storage to maintain high-throughput sequential disk operations. Independent consumer groups manage their own read positions, and an asynchronous replication protocol ensures high availability by allowing follower nodes to pull data without blocking primary write paths. Beyond core streaming, the system supports event-driven microservices, log aggregation, and archiving. It employs zero-copy network transfers to minimize overhead and provides a pluggable storage engine interface to accommodate various hardware configurations. Comprehensive documentation and API references are available to support integration and system management.

Debezium is a distributed change data capture platform that streams row-level database modifications as real-time events. By parsing database transaction logs, the system broadcasts structural and data changes to message brokers, enabling reactive processing and data integration across distributed architectures. The platform utilizes log-based capture to extract modifications directly from transaction logs, ensuring minimal impact on source system performance while maintaining the original commit order of operations. It employs database-specific connector adapters to translate proprietary binary formats into a unified event structure, supported by schema-registry-backed serialization to maintain consistent data definitions. To ensure a complete baseline for synchronization, the system performs snapshot-based initial states before transitioning to continuous event streaming. The tool supports a broad range of data integration tasks, including the maintenance of analytical stores and the synchronization of data across operational systems. Users can refine the data stream by applying filters to include or exclude specific tables, columns, or data types, and the system maintains an accurate representation of data models by parsing structural statements during the capture process. The project is implemented as a plugin for distributed message queues, facilitating integration into existing event-driven pipelines.

This project is a data processing engine and AI application platform designed for building production-grade machine learning workflows. It provides a unified programming model that handles both historical batch data and live stream ingestion, enabling the development of real-time ETL pipelines and scalable data transformation workflows. The framework distinguishes itself through differential dataflow execution, which propagates only changes through a pipeline rather than recomputing entire datasets. It supports distributed state management across worker nodes and utilizes incremental stream processing to trigger computations only when source data updates. These capabilities are paired with a specialized vector search framework that maintains low-latency access to evolving knowledge bases for retrieval-augmented generation. The platform facilitates enterprise AI integration by connecting large language models to private data sources. It includes pre-built application templates to assist in the deployment of high-accuracy retrieval systems and scalable data pipelines.

Electric is a Postgres data synchronization engine and replication proxy designed to enable local-first software. It replicates data from Postgres databases to client-side stores in real time using logical replication, allowing applications to maintain a local embedded database for offline access and low-latency updates. The system distinguishes itself by using shapes to filter and authorize specific subsets of database rows and columns before streaming them to clients or edge workers. It further supports multi-user collaboration by integrating a conflict-free replicated data type framework to ensure consistent state synchronization across different users. The project covers a broad range of capabilities, including reactive state management and real-time data streaming to client interfaces and server-side renders. It provides tools for data shaping and transformation, database integration across various cloud and serverless Postgres providers, and security primitives such as token-based authorization and end-to-end encryption. The service can be deployed as a containerized web service on cloud platforms with support for rolling deployment management.

Pathway is a high-performance data processing framework designed for building unified batch and streaming pipelines. It functions as an orchestrator for complex data transformations, utilizing a differential dataflow engine to process updates incrementally. By treating static datasets and continuous event streams with identical logic, the platform ensures exactly-once processing semantics and consistent results across diverse data sources. The framework distinguishes itself through its specialized support for real-time artificial intelligence and retrieval-augmented generation. It features integrated vector-aware data ingestion, which automates the creation and maintenance of searchable document indexes that update instantly as new data arrives. Developers can connect language models directly into their pipelines, utilizing built-in capabilities for document chunking, embedding generation, and result reranking to maintain synchronized, context-aware information retrieval. Beyond its core processing capabilities, the platform provides a robust infrastructure for deploying data applications. It supports the transition from batch to streaming workflows by simply updating input connectors, while its containerized deployment model allows for scaling services across local and cloud environments. The system is designed to handle large-scale event-driven tasks, providing a consistent programming model for both analytics and automated content generation workflows.

rqlite is a distributed relational database that replicates SQLite data across a cluster using the Raft consensus algorithm. It functions as a fault-tolerant storage system that provides high availability and a web API for executing SQL queries and managing relational data without requiring native database drivers. The system distinguishes itself by using an HTTP SQL interface to expose database operations and cluster management. It features a real-time change data capture stream that pushes database mutations to external HTTP endpoints via webhooks and supports the scaling of read throughput through non-voting read replicas. The project covers a broad range of distributed capabilities, including automated cluster discovery via DNS or Consul, TLS-encrypted transport for inter-node communication, and atomic request execution. It also includes tools for point-in-time snapshot backups, node health monitoring, and cluster leadership transfer.

RxJS is a library for reactive programming that provides a framework for composing asynchronous and event-based programs. It utilizes observable sequences to model data flows, allowing developers to manage complex sequences of events through a declarative programming interface. The library implements the observer pattern to facilitate decoupled communication between data producers and subscribers. By employing a lazy execution model, streams remain dormant until a consumer explicitly subscribes, at which point data production is triggered. This approach enables the construction of predictable, immutable data pipelines through functional operator chaining, which transforms and filters information as it moves through the system. The framework supports the coordination of concurrent data flows and asynchronous operations, including the management of backpressure to balance production and consumption rates. It provides tools for handling intricate user interface logic, such as debouncing inputs and synchronizing state with concurrent network requests, by merging and consolidating multiple event sequences into unified flows.

EventBus is a publish-subscribe messaging library designed to facilitate decoupled communication between components in Java applications. It functions as a central hub where producers dispatch events that are routed to subscribers based on the class type of the payload. By using annotation-based markers, the system maps event handlers to specific data types, allowing different parts of an application to exchange information without requiring direct references between classes. The library distinguishes itself through a focus on performance and execution control. It utilizes a compile-time indexing mechanism that generates static lookup tables, replacing slow runtime reflection with direct method calls to accelerate message routing. Furthermore, it provides a thread-aware dispatcher that allows developers to configure whether event handlers execute on the main interface thread, in background pools, or synchronously within the posting thread. Beyond basic routing, the system supports advanced messaging patterns including priority-ordered delivery and sticky events. Sticky events maintain a memory-based cache of recent data, ensuring that late-registering subscribers automatically receive the most current state upon initialization. The library also offers granular control over the event lifecycle, enabling developers to cancel event propagation or manage custom thread pools and error handling strategies to maintain application responsiveness.

Boto3 is the AWS SDK for Python, providing a programmatic interface for managing and automating AWS cloud infrastructure and services. It serves as a cloud management API client and resource manager for provisioning, configuring, and scaling virtual servers, databases, and storage. The library enables the implementation of infrastructure-as-code through declarative templates and scripts, allowing for the deployment of identical resource stacks across multiple accounts and geographic regions. It also provides a framework for coordinating distributed workflows, serverless functions, and containerized applications within the cloud ecosystem. The toolkit covers a broad range of operational capabilities, including generative AI orchestration, identity and access control, and detailed cloud resource monitoring. It further extends to data lifecycle management, including automated backups and migrations, as well as comprehensive billing and cost optimization tools.

NSQ is a distributed, brokerless messaging platform designed for high-throughput, fault-tolerant communication. By utilizing a decentralized topology, it eliminates single points of failure and allows for horizontal scaling across clusters. The system organizes message streams into topics and channels, effectively decoupling producers from consumers to support both streaming and job-oriented workloads. The platform distinguishes itself through a lookup-service-based discovery mechanism that enables clients to dynamically locate producers at runtime without requiring centralized coordination. To ensure reliability, it implements an explicit acknowledgement protocol that guarantees at-least-once message delivery, automatically re-queuing unhandled data. The system also manages memory usage by spilling message queues to disk when thresholds are exceeded, preventing service crashes during periods of high load. Beyond its core messaging capabilities, the project provides a comprehensive suite of administrative tools, including built-in HTTP endpoints for monitoring cluster health and managing configuration. It supports flexible deployment patterns, ranging from containerized environments to direct binary execution, and offers official client libraries alongside a documented TCP-based binary protocol for custom integrations. The software is available as pre-compiled binaries or source code, with documentation covering cluster administration, performance benchmarking, and operational configuration.

This project is a reactive, offline-first NoSQL database engine designed for JavaScript applications. It provides a robust framework for managing application state by synchronizing data across browsers, mobile devices, and server-side runtimes. By treating local storage as the primary source of truth, it enables applications to remain functional without network connectivity, automatically reconciling changes with remote backends once a connection is restored. The database distinguishes itself through a modular architecture that supports cross-environment synchronization and high-performance data management. It features a bidirectional replication protocol that handles conflict resolution and state convergence, alongside a pluggable storage abstraction that allows developers to swap between engines like IndexedDB, SQLite, or in-memory stores without altering application logic. To ensure responsiveness, the system offloads storage operations to background worker threads and coordinates database access across multiple browser tabs through a leader election mechanism. The platform offers a comprehensive suite of capabilities for data integrity, performance, and security. It enforces strict data validation through schema-based definitions and optimizes storage footprints using transparent key compression. Developers can bind database query results directly to user interface components, enabling reactive state management where the UI automatically updates in response to local or remote data changes. The project is built for extensibility, offering a wide range of plugins for encryption, full-text search, and integration with various backend protocols including GraphQL, REST, and peer-to-peer channels. It provides extensive documentation and standardized interfaces to facilitate integration into diverse application architectures.

Celery is an asynchronous job processor and distributed task queue designed to offload time-consuming operations to background worker nodes. By utilizing a message-passing architecture, it decouples task producers from consumers, allowing applications to maintain responsiveness while scaling workloads across multiple isolated environments. The system functions as a distributed workload orchestrator that manages the lifecycle of deferred operations through persistent queues. It distinguishes itself by providing a pluggable transport abstraction, which allows the core task logic to remain independent of specific messaging protocols. Furthermore, the framework includes built-in support for scheduled job execution, enabling the automation of recurring or delayed tasks without manual intervention. The platform also incorporates an event-driven monitoring framework that broadcasts internal system signals to provide real-time visibility into task lifecycles and worker node health. This diagnostic layer, combined with result-backend persistence and serialization-based payload management, ensures reliable task completion and consistent data transmission across distributed systems.

Arroyo is a high-performance stream processing platform built in Rust. It executes continuous SQL queries on streaming data with event-time semantics, enabling accurate windowed aggregations, joins, and stateful computations on unbounded event streams. The platform uses native Rust execution for high throughput and low latency, with periodic checkpointing for exactly-once fault tolerance and horizontal scaling across distributed workers. The system integrates deeply with Kafka for reading and writing topics with exactly-once delivery and supports change data capture (CDC) from MySQL and Postgres databases via Debezium. A wide range of source and sink connectors covers systems such as Kinesis, Redis, Delta Lake, Iceberg, MQTT, NATS, and more. SQL pipelines can be defined ad hoc or as derived streams, with support for user-defined functions written in Rust or Python for custom transformation logic. Deployment is managed through a web UI, CLI, and REST API, with options for single-node, multi-node, or Kubernetes clusters using Helm. Event-time processing includes watermarking to handle out-of-order data and supports tumbling, sliding, and session windows. The engine provides comprehensive SQL functions for string manipulation, timestamp arithmetic, JSON and array operations, data type conversion, and mathematical computations. Additional operational features include anomaly detection by counting events over time windows, synthetic data generation for testing, and authentication and TLS encryption for secure access.

TDengine is a distributed time-series database designed for the high-speed ingestion, compression, and retrieval of timestamped metrics and sensor data. It functions as a SQL-compatible analytics engine, allowing users to perform complex operations on massive volumes of time-ordered information using standard relational syntax. The platform is built to serve as a backend foundation for industrial IoT environments, managing real-time data streams and device metadata through a cluster-based architecture. The system distinguishes itself through a distributed sharding architecture that uses consistent hashing to ensure horizontal scalability and high-throughput ingestion. It employs a log-structured write path to minimize disk seek latency and utilizes super-table virtualization to provide a unified logical view across multiple physical tables. To maintain performance and cost-efficiency, the database features automated multi-tiered lifecycle management, which migrates data between high-performance memory and low-cost storage based on age and access frequency. Beyond its core storage capabilities, the platform provides robust tools for edge-to-cloud synchronization, ensuring consistent data states across geographically distributed infrastructure. It includes built-in support for real-time stream processing, allowing for the analysis of live data without requiring external message queues. The system also incorporates comprehensive security frameworks, including user access control, audit logging, and encrypted transport protocols to protect sensitive operational data. Developers can interact with the database through native client libraries that support connection pooling and query parameter binding. The system is documented with comprehensive error code diagnostics and provides command-line utilities for cluster administration, health monitoring, and configuration management.