timgit/pg-boss
Pg Boss
pg-boss is a background task scheduler and distributed task queue that uses PostgreSQL as a reliable message broker for asynchronous job processing. It provides a system for distributing work across multiple application instances, ensuring exactly-once delivery through atomic database transactions.
The project includes a cron job scheduler for automating recurring tasks and a PostgreSQL pub-sub system for fan-out event distribution. It also features a web-based management dashboard for monitoring queue statistics and controlling job lifecycles, including manual retries and cancellations.
Capabilities cover job lifecycle management with state-machine tracking and exponential backoff retries. The system supports queue storage partitioning for high-volume workloads and provides a command-line interface for managing database schema migrations.
The management interface includes basic authentication for dashboard and API security, with support for custom base path configurations to allow hosting behind reverse proxies.
Features
- Background Job Queues - Uses PostgreSQL as a reliable message broker to provide a background job queue with exactly-once delivery.
- Job Queues - Provides a reliable background job queue powered by PostgreSQL for asynchronous task processing.
- Advisory Locking Mechanisms - Utilizes database-level locking to maintain distributed locks across multiple application instances.
- Cron Scheduling - Implements a cron job scheduler that uses a relational database to track the state of recurring tasks.
- Recurring Job Scheduling - Implements a recurring task scheduler using cron expressions to automate periodic jobs.
- Distributed Job Locking - Implements high-concurrency job fetching using the FOR UPDATE SKIP LOCKED SQL pattern.
- Distributed Task Queues - Distributes asynchronous tasks across multiple worker instances using a shared relational database.
- Reliable Task Queues - Ensures reliable job processing with exactly-once delivery guarantees and automated retry logic.
- Database Job Lifecycle Management - Manages the full lifecycle of jobs, including status transitions and manual retries via the database.
- Transactional Enqueueing - Allows job creation within a database transaction to ensure atomic updates between state and queue.
- Task Throughput Rate Limiters - Implements rate limiting and concurrency controls to manage job processing throughput.
- Cross-Application Event Pub-Sub - Implements fan-out queue relationships that broadcast a single event to multiple subscribers.
- PubSub Messaging Systems - Implements a pub-sub system using the database to notify multiple subscribers of a single event.
- Exponential Backoff Retries - Employs an exponential backoff strategy for failed job retries to prevent database overload.
- Fan-Out Event Distribution - Provides a pub-sub system that distributes a single event across multiple queues for fan-out notification.
- Management Dashboards - Provides a web interface for monitoring queue statistics and manually retrying or cancelling jobs.
- Queue Visualizers - Offers a visual interface to display queue statistics and job payloads for state inspection.
- Pub/Sub Event Distribution - Implements a pub-sub system that distributes a single event across multiple destination queues.
- Partitioned - Optimizes high-volume workloads by partitioning queues into dedicated physical database tables.
- Retry State Tracking - Tracks job progress through a set of predefined states to manage retries and failures.
- Background Job Monitoring - Ships a visual dashboard for tracking the status, health, and failure metrics of background task queues.
- Queue Administration Interfaces - Provides a web-based administrative interface for monitoring and managing background job queues.
- Web Dashboards - Provides a centralized web-based dashboard for tracking the state of jobs, queues, and schedules.
- Job Queues - Node.js job queueing system.
