antirez/disque
Disque
Disque is a distributed message broker that manages asynchronous communication between services through in-memory job queues. It operates as a cluster-aware job scheduler, providing at-least-once delivery guarantees with acknowledgment and retry mechanisms, and supports blocking queue consumers that efficiently wait for jobs to become available.
The broker implements a job state machine that tracks each job through its lifecycle from queued to acknowledged, with configurable retry limits and time-to-live expiration to prevent unbounded queue growth. Jobs can be enqueued with delivery guarantees, fetched via blocking or non-blocking calls, acknowledged upon completion, and rejected for reprocessing. The system supports extending job processing time to prevent premature redelivery, and queue operations can be paused and resumed with cluster-wide broadcast for coordinated maintenance.
Disque uses an asynchronous replication model where jobs are replicated to slave nodes without waiting for confirmation, prioritizing low latency over strong consistency. Nodes coordinate through a gossip-based protocol for peer discovery and state propagation without a central coordinator. All job data resides in memory for fast access, with optional persistence to disk for recovery after restarts. The broker provides queue and job state inspection capabilities, allowing operators to view queue length, job age, blocked clients, and individual job details.
Features
- At-Least-Once Delivery Guarantees - Ensures every job is processed at least once using acknowledgments and redelivery.
- In-Memory Queues - Stores all job data in memory for fast access with optional disk persistence.
- Asynchronous Replication Management - Implements asynchronous replication where master acknowledges writes before all replicas confirm.
- Lifecycle State Machines - Implements a job state machine tracking each job through its lifecycle from queued to acknowledged.
- Retry and TTL Configurations - Adds jobs to queues with configurable retry limits, time-to-live, and delivery delays.
- Gossip Protocols - Uses gossip-based protocol for peer discovery and state propagation across the cluster.
- Message Brokers - Distributes jobs across multiple nodes in a cluster for fault-tolerant message processing.
- Queue Pause and Resume Controls - Provides queue pause and resume controls with cluster-wide broadcast for coordinated maintenance.
- Acknowledgments - Signals successful job completion to remove it from the queue and prevent redelivery.
- Job Expirations - Automatically evicts jobs after configurable time-to-live to prevent unbounded queue growth.
- Processing Time Extensions - Provides a mechanism for workers to extend job processing time, preventing premature redelivery.
- Job Rejections and Requeues - Implements job rejection with automatic requeue and retry counter for at-least-once delivery.
- Manual Job Fetching - Ships a blocking and non-blocking job fetch capability for retrieving jobs from queues.
- Blocking Fetches - Provides blocking job fetch that waits until a job is available or a timeout expires.
- Blocking Consumers - Provides blocking queue consumers that efficiently wait for jobs without polling overhead.
- Message Brokers - Manages asynchronous communication between services through in-memory job queues.
- Enqueue Operations - Adds messages to queues with configurable replication, delay, retry, and time-to-live settings.
- Queue Consumers - Provides blocking queue consumers that efficiently wait for jobs without busy-waiting.
- Retry Counters - Tracks retry counts on rejection to limit redeliveries and discard repeatedly failing jobs.
- Inspections - Provides queue and job state inspection for viewing length, age, blocked clients, and job details.
- Queue and Job State Inspections - Offers inspection commands for queue length, job age, blocked clients, and individual job details.
