OptimalBits/bull
Bull
Bull is a Node.js library for managing distributed jobs and message queues using Redis as the primary data store. It functions as a distributed task worker, job scheduler, and priority queue manager designed to handle asynchronous workloads across multiple processes.
The project distinguishes itself by providing a persistent communication channel that decouples servers through the exchange of serializable data objects. It ensures distributed system reliability by detecting stalled tasks and recovering from process crashes to ensure every queued job is completed.
The system covers a broad range of queue management capabilities, including priority-based task ordering, automatic retry policies, and delayed or recurring job execution via cron specifications. It provides observability tools for tracking job progress, querying execution states, and monitoring queue events. To maintain performance, it supports task concurrency scaling, rate limiting, and child-process sandboxing for CPU-intensive workloads.
The library includes specific integrations for Redis environments, such as connection pooling and hash-slot key prefixing for compatibility with Redis clusters.
Features
- Distributed Task Queues - Implements a distributed task queue to process background work across multiple nodes reliably and at scale.
- Redis-Backed Queues - Provides a distributed task queue that uses Redis as the primary data store for persistent state and message brokering.
- Cron Scheduling - Supports automated job execution based on cron expressions for recurring background tasks.
- Job Schedulers - Acts as a job scheduler for executing asynchronous functions with support for delayed and cron-based execution.
- Background Task Runners - Orchestrates the lifecycle of heavy background tasks with custom priority, retry policies, and concurrency limits.
- Task Schedulers - Automates periodic maintenance and updates using cron specifications to trigger recurring background jobs.
- Distributed Processing - Distributes asynchronous task processing across multiple Node.js worker processes using a shared Redis backend.
- Distributed Task Workers - Functions as a distributed task worker that processes heavy workloads across multiple threads or processes.
- Job Priority Management - Manages job execution order based on priority levels, concurrency limits, and automatic retry policies.
- Message Queues - Implements a persistent message queue for asynchronous task buffering and reliable event-driven workflows.
- Heartbeat Monitors - Tracks worker liveness through periodic heartbeats to automatically detect and re-queue stalled jobs.
- Job Queues - Provides infrastructure for managing asynchronous task execution by enqueueing serializable data objects.
- Job Processors - Provides an engine to define job processors with associated concurrency limits and routing rules.
- Message Queuing Architectures - Implements a message queuing architecture to decouple servers via asynchronous, persistent communication channels.
- Persistent Operation Queues - Persists queue metadata and job payloads in Redis to ensure reliability across process restarts.
- Reliable Task Queues - Ensures distributed system reliability by recovering from process crashes and automatically handling stalled tasks.
- Task Retry Policies - Implements automatic job retries based on defined policies to recover from transient execution errors.
- Lua Scripting - Utilizes server-side Lua scripts to perform atomic state transitions and prevent race conditions between distributed workers.
- State Querying - Allows querying of specific job instances and aggregate counts of jobs in various execution states.
- Slot-Based Routing - Uses hash-slot key prefixing to ensure all keys for a queue map to the same Redis cluster shard for atomicity.
- Crash Recovery Systems - Automatically recovers from process crashes by detecting and restarting interrupted jobs.
- Delayed Executions - Provides the capability to postpone the execution of background tasks until a specified future time.
- Process Sandboxing - Executes CPU-intensive jobs in separate processes to prevent main-thread crashes and event-loop blocking.
- Lifecycle Management - Enables manual control over job lifecycles, including retrying failed tasks and promoting delayed ones.
- Hanging Job Alerts - Detects hanging processes through heartbeat monitoring and automatically triggers alerts or retries.
- Polling-Based Worker Execution - Employs a polling-based execution model where workers retrieve tasks using Redis lists and sorted sets.
- Process Scaling - Scales throughput by running multiple jobs simultaneously across independent worker processes.
- Worker Throttling - Implements processing rate controls to prevent background jobs from overloading downstream dependencies.
- Message Brokers - Functions as a Redis-based message broker to decouple servers through the exchange of serializable data objects.
- Execution Control - Offers mechanisms to pause or resume job processing globally across all workers or locally.
- Job Process Sandboxing - Provides child-process sandboxing to execute CPU-intensive or unstable jobs without blocking the main Node.js event loop.
- Rate Limiting - Caps the number of jobs processed within a specific timeframe across all distributed workers to maintain system stability.
- Task Sequencing Strategies - Supports flexible job processing sequences including first-in-first-out, last-in-first-out, and priority-based ordering.
- Event Monitoring - Provides event monitoring to notify the system when jobs change state, such as completing or failing.
- Job Monitoring Tools - Provides tools to track job progress via completion percentages and custom log history.
- Job Queues - Persistent job and message queue system.
- Messaging and Event Streaming - Fast and reliable Redis-based queue for Node.js applications.
- Task Queues - Persistent job and message queue.
