Redpanda

Redpanda is a distributed event streaming engine designed to serve as a high-performance, drop-in replacement for existing event-driven architectures. It provides a foundation for building and scaling applications that require reliable data movement, analytical querying, and strict operational compliance across both cloud and self-managed environments.

The platform distinguishes itself through a shared-nothing architecture that utilizes thread-per-core execution and a non-blocking asynchronous input/output engine to maximize throughput. It maintains data consistency through a consensus-based replication model and implements binary protocol compatibility, allowing existing ecosystem tools to interact with the system without modification. To optimize resource usage, the platform features a zero-copy data path and automated tiered storage that offloads historical log segments to object storage while maintaining a unified view for consumers.

Beyond core streaming, the platform includes integrated governance and orchestration capabilities for connecting autonomous agents to data flows. It provides granular identity management and execution controls to secure agent interactions, alongside auditing tools that record immutable logs of system actions. The infrastructure also supports real-time analytical querying across live and historical data streams to facilitate immediate operational insights.

Features

Data Streaming Platforms - Acts as a high-performance, drop-in replacement for Kafka to simplify event-driven operations.
Distributed Event Streaming Platforms - Provides a high-throughput distributed engine for processing real-time data streams.
Real-Time Analytics - Executes analytical queries across live and historical data streams to provide immediate operational insights.
Real-Time Data Streaming - Transmits real-time data streams between disparate systems to facilitate low-latency communication.
Kafka-Compatible Deployments - Provisions high-performance clusters that support existing event-driven applications with Kafka compatibility.

hazelcast/hazelcast

6,570View on GitHub

Hazelcast is a distributed data platform that combines an in-memory data grid with a stream processing engine to support real-time analytics and event-driven applications. It functions as a partitioned, distributed key-value store that replicates data across cluster nodes to provide low-latency access and high availability. The platform also serves as a distributed SQL query engine, allowing users to execute standard SQL statements against both in-memory datasets and external data sources. What distinguishes Hazelcast is its use of a distributed consensus subsystem to maintain strongly consis

apache/incubator-rocketmq

22,461View on GitHub

RocketMQ is a distributed messaging and streaming platform designed for building event-driven applications. It serves as middleware to decouple services using publish-subscribe and request-reply patterns, and functions as a transactional messaging system that ensures atomicity by linking message delivery to local transaction outcomes. The platform includes specialized capabilities as a Kubernetes-native message broker for container orchestration environments and an MQTT broker for ingesting event data from mobile applications and hardware terminals. The system covers high-throughput data str

apache/kafka

32,846View on GitHub

Kafka is a distributed event streaming platform designed for capturing, storing, and processing real-time data streams across interconnected nodes. It functions as a distributed commit log, providing a fault-tolerant storage mechanism that records state changes sequentially to ensure data consistency and durability across distributed environments. The platform distinguishes itself through a partitioned commit log architecture that enables horizontal scaling and parallel processing of data streams. It integrates a stream processing engine for continuous transformations and aggregations, while

PrefectHQ/prefect

21,640View on GitHub

Prefect is a workflow orchestration platform designed to define, schedule, and monitor complex data pipelines as Python code. It functions as a container-native engine that wraps individual tasks in isolated environments, ensuring consistent dependencies and resource allocation across diverse infrastructure. By utilizing a state-machine-based orchestration model, the system tracks execution progress through discrete transitions and persistent event logs to maintain reliable and observable task processing. The platform distinguishes itself through a decoupled worker-API architecture, which sep

redpanda-dataredpanda

Features