apache/pulsar

Features

• Pub-Sub Messaging - Functions as a cloud-native distributed pub-sub messaging system that decouples producers and consumers for high-performance data ingestion.
• Cursor-Based Change Streaming - Maintains individual consumer offsets using cursors to enable reliable message replay and independent stream tracking.
• Regional Replication - Synchronizes message sequences across geographically distinct regions to ensure high availability and disaster recovery.
• High-Throughput Ingestion Pipelines - Designed to ingest massive volumes of incoming data through a distributed architecture with decoupled compute and storage.
• Horizontal Scaling - Expands system capacity across multiple nodes using a layered architecture that avoids data reshuffling during scaling.
• Message Ordering Management Systems - Directs messages to specific partitions via key hashing to maintain strict ordering while scaling throughput.
• Event Stream Isolation - Provides a multi-tenant platform that isolates event streams and resources via authentication and storage quotas.
• Multi-Tenant Resource Isolation - Isolates workloads for different organizations through authentication and resource quotas on a shared cluster.
• Stream Computing Engines - Provides a lightweight compute engine to transform and route events in real-time directly on the data stream.
• Storage-Compute Architectures - Decouples the serving layer from the storage layer to allow independent scaling of compute and data retention.
• Storage Tiering - Offloads historical messaging data from high-performance disks to low-cost cloud storage to optimize long-term retention.
• Storage Decoupling - Implements an architecture that separates serving layers from storage layers to allow independent scaling of compute and retention.
• Consumer Cursor Tracking - Maintains consumer cursor positions to ensure messages are processed reliably from the correct point in the stream.
• Topic Distribution - Automatically distributes topics across brokers and splits bundles to ensure even resource utilization.
• Message Stream Replayers - Allows consumers to reset to a specific message ID or timestamp to re-process historical data.
• Message Ordering Guarantees - Guarantees that messages are processed in the exact order they were produced to maintain strong consistency.
• Topic Sharding - Distributes topic ownership across brokers using a bundle-splitting mechanism to optimize cluster load and scalability.
• Retention Policy Management - Defines how long consumed messages are kept before deletion based on time or size limits.
• Stream Processing Engines - Implements a compute framework for continuous, low-latency transformation and analysis of high-velocity data streams.
• Serverless Processing - Provides a serverless engine to execute compute functions directly on streams for data transformation and routing.
• Atomic Transactions - Coordinates atomic operations across multiple topics to ensure data consistency during complex streaming workflows.
• External System Synchronization - Provides a pluggable architecture for bi-directional data streaming between the platform and external systems via source and sink connectors.
• Pluggable Connector Frameworks - Features a modular connector framework to integrate heterogeneous external data sources and sinks.
• Disk Quota Enforcement - Enforces storage limits per tenant and blocks producers when quotas are reached to prevent resource exhaustion.
• Backpressure Controllers - Implements flow control mechanisms to regulate data ingestion and prevent system memory overflow.
• REST Administrative APIs - Provides a REST-based administrative API for provisioning resources and managing cluster state.
• Data Ingestion - Distributed pub-sub messaging platform with flexible messaging models.
• Data Ingestion Pipelines - Distributed pub-sub messaging platform with flexible messaging models.
• Data Engineering - Cloud-native distributed messaging and streaming platform.

Star history