apache/pulsar
Pulsar
Apache Pulsar is a cloud-native distributed pub-sub messaging system designed for high-performance data ingestion. It functions as a geo-replicated data streamer and a multi-tenant event streaming platform, providing a serverless stream processing engine and a tiered storage messaging broker.
The system distinguishes itself by separating serving layers from storage layers to allow independent scaling of compute and data retention. It features native geo-replication to synchronize messages across different geographical regions and employs a multi-layered tenant isolation model using authentication and storage quotas to support multiple organizations on a single cluster.
The platform provides capabilities for atomic transaction management, message offset replay, and strict message ordering guarantees. Its operational surface includes a pluggable connector framework for external system connectivity, tiered storage for offloading historical data, and a REST interface for cluster management and resource provisioning.
The project provides official containerized deployment images and supports horizontal infrastructure scaling.
Features
- Pub-Sub Messaging - Functions as a cloud-native distributed pub-sub messaging system that decouples producers and consumers for high-performance data ingestion.
- Cursor-Based Change Streaming - Maintains individual consumer offsets using cursors to enable reliable message replay and independent stream tracking.
- Regional Replication - Synchronizes message sequences across geographically distinct regions to ensure high availability and disaster recovery.
- High-Throughput Ingestion Pipelines - Designed to ingest massive volumes of incoming data through a distributed architecture with decoupled compute and storage.
- Horizontal Scaling - Expands system capacity across multiple nodes using a layered architecture that avoids data reshuffling during scaling.
- Message Ordering Management Systems - Directs messages to specific partitions via key hashing to maintain strict ordering while scaling throughput.
- Event Stream Isolation - Provides a multi-tenant platform that isolates event streams and resources via authentication and storage quotas.
- Multi-Tenant Resource Isolation - Isolates workloads for different organizations through authentication and resource quotas on a shared cluster.
- Stream Computing Engines - Provides a lightweight compute engine to transform and route events in real-time directly on the data stream.
- Storage-Compute Architectures - Decouples the serving layer from the storage layer to allow independent scaling of compute and data retention.
- Storage Tiering - Offloads historical messaging data from high-performance disks to low-cost cloud storage to optimize long-term retention.
- Storage Decoupling - Implements an architecture that separates serving layers from storage layers to allow independent scaling of compute and retention.
- Consumer Cursor Tracking - Maintains consumer cursor positions to ensure messages are processed reliably from the correct point in the stream.
- Topic Distribution - Automatically distributes topics across brokers and splits bundles to ensure even resource utilization.
- Message Stream Replayers - Allows consumers to reset to a specific message ID or timestamp to re-process historical data.
- Message Ordering Guarantees - Guarantees that messages are processed in the exact order they were produced to maintain strong consistency.
- Topic Sharding - Distributes topic ownership across brokers using a bundle-splitting mechanism to optimize cluster load and scalability.
- Retention Policy Management - Defines how long consumed messages are kept before deletion based on time or size limits.
- Stream Processing Engines - Implements a compute framework for continuous, low-latency transformation and analysis of high-velocity data streams.
- Serverless Processing - Provides a serverless engine to execute compute functions directly on streams for data transformation and routing.
- Atomic Transactions - Coordinates atomic operations across multiple topics to ensure data consistency during complex streaming workflows.
- External System Synchronization - Provides a pluggable architecture for bi-directional data streaming between the platform and external systems via source and sink connectors.
- Pluggable Connector Frameworks - Features a modular connector framework to integrate heterogeneous external data sources and sinks.
- Disk Quota Enforcement - Enforces storage limits per tenant and blocks producers when quotas are reached to prevent resource exhaustion.
- Backpressure Controllers - Implements flow control mechanisms to regulate data ingestion and prevent system memory overflow.
- REST Administrative APIs - Provides a REST-based administrative API for provisioning resources and managing cluster state.
- Data Ingestion - Distributed pub-sub messaging platform with flexible messaging models.
- Data Ingestion Pipelines - Distributed pub-sub messaging platform with flexible messaging models.
- Messaging Systems - Distributed messaging and streaming platform.
- Data Engineering - Cloud-native distributed messaging and streaming platform.
