taskforcesh/bullmq

Features

• Background Job Processing - Provides a robust system for offloading heavy computations to asynchronous background tasks to prevent event loop blocking.
• Distributed Job Execution - Processes tasks across a fleet of distributed workers using Redis to scale throughput.
• Distributed Task Schedulers - Schedules delayed and recurring jobs with support for cron expressions and automatic retry strategies.
• Job Schedulers - Automates recurring or delayed operations using cron expressions and fixed intervals via Redis.
• Distributed Locks - Prevents concurrent execution of the same job by locking the task during its processing lifecycle.
• Process-Based Isolation - Executes job processors in separate child processes to isolate heavy workloads and prevent event loop blocking.
• Execution Rate Limiters - Caps the number of jobs processed over a specific time window to prevent overloading external systems.
• Job Concurrency Controllers - Manages and dynamically updates the number of jobs processed simultaneously to prevent resource exhaustion.
• Sequential Workflows - Implements sequential job execution where steps are triggered only after the previous task completes.
• Job Priority Management - Organizes jobs into different priority levels to control the execution order over a standard FIFO sequence.
• Job Queues - Provides the core mechanism to insert units of work into queues for asynchronous distributed processing.
• Job Scheduling - Provides comprehensive management of job states, including transitions to completed and priority adjustments.
• Job Dependency Coordination - Orchestrates complex multi-step workflows by defining dependencies between parent and child jobs.
• Job Event Callbacks - Triggers automatic execution of custom functions when jobs are completed, failed, or updated.
• State-Aware Job Orchestrators - Coordinates multi-step workflows by managing complex job dependencies and parent-child hierarchies.
• Message Queues - Provides persistent message queues to decouple services and ensure delivery when receivers are offline.
• Rate Limiting - Controls the frequency of job processing using time-windowed counters to protect downstream services from overload.
• Distributed Coordination Services - Coordinates the production and consumption of messages across multiple distributed services using a shared Redis store.
• Asynchronous Background Processors - Offloads time-consuming operations to isolated background processes to maintain main application responsiveness.
• Distributed Job Locking - Locks jobs during processing to ensure that only one worker handles a specific task at a time.
• Distributed Task Queues - Manages asynchronous workloads across multiple nodes using Redis to ensure reliable job delivery and scalability.
• Redis-Backed Queues - Uses a shared Redis data store to maintain job queues and state across distributed worker nodes.
• Job Process Sandboxing - Executes job processors in separate child processes to isolate failures and prevent event loop blocking.
• Parent-Child Dependencies - Coordinates complex workflows by blocking parent jobs until their associated child jobs reach a final state.
• Rate Limiting - Controls the execution frequency and concurrency of background tasks to protect downstream services from overload.
• Task Retry Policies - Provides automatic retry strategies and state recovery to ensure reliable execution after transient failures or crashes.
• Workflow Orchestrators - Coordinates complex multi-step distributed processes by defining parent-child job dependencies and hierarchies.
• Atomic Batch Operations - Inserts multiple jobs across different queues in a single atomic step to maintain consistency.
• Lua Scripting - Executes complex queue transitions and state updates via Lua scripts to ensure atomicity within Redis.
• Exactly-Once Processing Semantics - Ensures jobs are processed exactly once by tracking states from creation to completion.
• Hash Tag Atomicity - Ensures atomic operations across keys by using hash tags in queue names within clustered environments.
• Cluster Connectivity - Connects to managed Redis databases using cluster mode to distribute queues across nodes.
• Queue Deletion - Wipes entire queues and their contents, including the option to force remove active jobs.
• Execution Time Limits - Automatically terminates sandboxed worker processes that exceed a specified execution time limit to prevent hanging.
• Graceful Shutdowns - Stops workers from accepting new jobs while allowing current tasks to complete before shutting down.
• Delayed Executions - Postpones the execution of a processing job and releases the worker lock for later retry.
• Timeout-Based Cancellation - Automatically stops job execution when a specified global or per-job time limit is reached.
• Graceful Queue Draining - Processes all remaining jobs in a queue until empty before performing a system shutdown.
• Job Deduplication - Prevents duplicate tasks from being enqueued using unique job identifiers.
• Queue Data Cleanup - Deletes completed or failed jobs based on limit or age to prevent Redis storage exhaustion.
• Queue Maintenance - Removes old jobs based on grace periods and promotes delayed jobs to active status.
• Queue Purging - Enables the clearance of waiting or delayed jobs from the queue.
• Atomic Job Flow Creation - Creates complex dependent job structures atomically to ensure consistency in workflow orchestration.
• Failure Propagation - Automatically propagates failure from a child job to its parent in a dependent workflow.
• Group Execution Control - Allows workers to stop picking up new jobs for specific job groups.
• Job Behavior Configurations - Allows configuring whether a parent job should continue execution if a child job fails.
• Job Cancellation - Terminates active processing jobs using abort signals to ensure proper resource cleanup.
• Stalled Job Recovery - Monitors the queue to identify stalled jobs and move delayed jobs to waiting for retry.
• Work Queue Performance Metrics - Records job completion and failure rates over intervals to provide historical performance data for the queue.
• Job Batching - Groups multiple background tasks together for a single worker to reduce network and processing overhead.
• Asynchronous Progress Tracking - Emits numeric or object updates during job execution to notify listeners of completion state.
• Concurrent Task Limiters - Limits the number of concurrent jobs that can be processed for a specific group to manage resource consumption.
• Event-Driven Callbacks - Triggers asynchronous callbacks based on job lifecycle transitions such as completion, failure, or progress updates.
• Execution Pausing - Provides controls to halt and restart job execution globally or for individual worker instances.
• Job Result Persistence - Sends job outcomes to dedicated queues for reliable downstream processing and persistent storage.
• Child Result Aggregation - Aggregates return values from child jobs to be utilized by the parent job in a workflow hierarchy.
• Multi-Step Job Tracking - Tracks job progress through internal sequences to enable resuming from the last successful step after a failure.
• Partitioned - Distributes workloads across multiple CPU cores by partitioning queues into keyed segments for higher throughput.
• Retry Policies - Calculates wait times between job retries using custom functions based on attempt count and error type.
• Background Job Monitoring - Monitors job progress and tracks the status and health of active asynchronous tasks.
• Metric Collection - Tracks quantitative telemetry including completion counts, failure rates, and processing durations.
• Distributed Tracing - Propagates trace context across services to track the complete lifecycle of jobs in distributed flows.
• Queue Depth Monitoring - Retrieves real-time counts of waiting and delayed jobs to monitor queue depth and system health.
• Prometheus Exporters - Provides a scrapeable endpoint to export performance metrics in the Prometheus format for external monitoring.
• Queue Volume Monitoring - Retrieves the total or active count of jobs within a specific group.
• Job Queues - Persistent job and message queue for Node.js.
• Task Queues - Advanced message queue based on Redis.

Star history