apache/incubator-pulsar
Incubator Pulsar
Apache Pulsar is a cloud-native message queue and distributed publish-subscribe messaging system. It serves as a multi-tenant event streaming platform designed to route data streams for asynchronous communication between producers and consumers.
The system distinguishes itself through geo-replication, synchronizing data across multiple geographic regions to ensure high availability and low latency. It implements a multi-tenant architecture that provides isolation and resource management for millions of independent topics.
The platform covers high-throughput data streaming and event-driven data pipelines. Its capabilities include maintaining message ordering, managing multi-tenant access, and offloading cold data to tiered storage. System administration and resource provisioning are handled via a programmatic REST API.
Features
- Message Queues - Provides a cloud-native, scalable message queue that separates storage from serving for independent scaling.
- Publish-Subscribe Messaging - Implements a distributed publish-subscribe model for asynchronous communication between producers and consumers.
- Queue and Messaging - Provides reliable message queueing semantics to ensure data delivery across a distributed cluster.
- Event-Driven Data Pipelines - Provides reactive systems that route and buffer messages in a reliable sequence between distributed components.
- Regional Replication - Implements synchronization of message data across geographically distinct locations to ensure global availability and disaster recovery.
- High-Throughput Data Streaming - Optimizes the continuous flow and processing of massive volumes of event data across millions of topics.
- Event Stream Isolation - Provides built-in isolation and resource management for millions of independent topics across multiple tenants.
- Multi-Tenant Resource Isolation - Groups topics into namespaces with specific policies for quotas and security to provide strong multi-tenant resource isolation.
- Message Queues - Stores messages in a reliable buffer to manage event delivery between distributed components without data loss.
- Pub-Sub Messaging - Decouples message producers and consumers using topics and channels for asynchronous communication.
- Multi-Tenant Identity Management - Isolates user accounts and data within shared environments using authentication, authorization, and resource limits.
- Messaging Isolation Fabrics - Enforces multi-tenant data separation and controlled stream sharing within the messaging infrastructure.
- Message Data Offloading - Moves older messaging data from high-speed local disks to remote object storage to extend retention periods.
- Write Acknowledgement Strategies - Ensures data durability by requiring a majority of storage nodes to acknowledge a write before completion.
- Messaging Throughput Scaling - Dynamically adjusts resources to handle massive message volumes across millions of topics.
- Segmented Storage Architectures - Splits message logs into smaller segments distributed across a cluster for independent scaling of storage and serving.
- Message Ordering Guarantees - Ensures messages arrive at the consumer in the exact sequence they were sent to preserve data consistency.
- Stateless Serving Layers - Separates the stateless serving layer from the stateful storage layer to enable independent scaling of compute and disk.
- Data Pipelines - Distributed pub-sub messaging platform.
- Message Queues - Cloud-native distributed messaging and streaming platform.
