pgmq/pgmq
Pgmq
PGMQ ist ein leichtgewichtiges Message-Queuing-System, das als PostgreSQL-Erweiterung implementiert ist, um asynchrone Aufgaben zu verwalten. Es fungiert als datenbankgestützter Message-Broker, der PostgreSQL für persistente Speicherung, atomare Operationen und benachrichtigungsgesteuerte Zustellung nutzt.
Das System bietet ein SQS-kompatibles Queue-Modell mit Sichtbarkeits-Timeouts und verzögerter Zustellung. Es unterstützt eine strikte First-in-First-out-Reihenfolge durch Gruppen-Keys und Batch-Abruf, um die sequentielle Verarbeitung verwandter Aufgaben sicherzustellen.
Das Projekt deckt den gesamten Lebenszyklus einer Nachricht ab, einschließlich Produktion, Konsum über atomare Pop-Operationen und administrative Verwaltung wie Queue-Bereinigung und Lebenszykluskontrolle. Es enthält Routing-Funktionen durch themenbasierte Muster und Zuverlässigkeitsmerkmale wie Dead-Letter-Queueing, Nachrichtenarchivierung und Wiederholungslogik. Überwachungstools werden bereitgestellt, um operative Metriken wie Queue-Länge und Durchsatz zu verfolgen.
Features
- Postgres Message Queueing - Implements a full asynchronous task processing system as a PostgreSQL extension using the database as the broker.
- Database-Backed Brokers - A broker that leverages PostgreSQL for persistent storage, atomic operations, and notification-driven message delivery.
- PostgreSQL Task Queues - Provides a lightweight message queuing system implemented as a PostgreSQL extension to manage asynchronous tasks.
- Distributed Task Workers - Manages the delivery and consumption of tasks across multiple distributed worker processes.
- Atomic Queue Transitions - Provides atomic operations to retrieve and delete messages in a single step to ensure at-most-once delivery.
- Message Queues - Provides the primary mechanism for adding data payloads to a queue with optional delivery delays.
- Message Consumption - Implements message retrieval mechanisms including long-polling and visibility timeouts to manage consumer access.
- Message Deletion Operations - Provides capabilities to remove single or batch messages from the queue after successful processing.
- Table-Based Storage - Uses dedicated PostgreSQL tables to store message payloads and metadata for persistent queue management.
- FIFO - Provides a queueing system that ensures strict first-in-first-out ordering of messages using group keys.
- Message Processing Timeouts - Implements visibility timeouts to prevent premature message redelivery while a task is being processed.
- Visibility Timeout Controls - Provides granular control over message visibility timeouts to manage processing attempts and redelivery.
- Atomic Pop Operations - Implements atomic pop operations that combine message retrieval and deletion in one transaction for at-most-once delivery.
- Skip Locked Concurrency - Employs the FOR UPDATE SKIP LOCKED clause to allow multiple consumers to process messages without blocking each other.
- FIFO Group Processing - Supports strict first-in-first-out ordering for related tasks using group keys to prevent race conditions.
- FIFO Task Sequencing - Ensures strict order of operations for related tasks using group identifiers to prevent race conditions.
- Group-Keyed FIFO Ordering - Maintains strict processing sequences for related messages using group keys and index-based sorting.
- Database Insert Notifications - Employs Postgres NOTIFY events to alert consumers of new messages immediately without requiring constant polling.
- Queue Purging - Ships a dedicated operation to clear all pending messages from a queue without deleting the queue itself.
- Dead Letter Queues - Moves repeatedly failing messages to a separate dead-letter queue to prevent processing blocks.
- Queue-Specific Metrics - Provides detailed operational statistics including queue length, message age, and overall throughput.
- Message Requeuing - Implements the ability to re-inject failed messages into the queue by resetting their visibility timeouts.
- Queue Lifecycle Management - Implements administrative tools for creating and deleting the database tables used for queue storage.
- Topic Routing Patterns - Implements flexible message distribution using topics and wildcard patterns for complex data delivery.
- SQS-Compatible Queues - Offers an SQS-compatible queue model featuring visibility timeouts and delayed delivery.
- Database Trigger Notifications - Uses Postgres NOTIFY events to alert consumers of new message inserts without requiring constant polling.
- Queue Throughput Analytics - Provides tools for tracking queue length and message throughput to monitor background processing health.
- Job Queues - Lightweight message queue built on PostgreSQL.
