bee-queue/bee-queue
Bee Queue
Bee-queue is a Node.js background processing system that uses Redis for job queueing and persistence. It is designed to offload heavy tasks from the main execution thread to background workers to maintain application responsiveness.
The project provides distributed job processing, allowing worker nodes to run across multiple processes to handle large volumes of tasks concurrently. It ensures reliable task execution through automatic retries and the recovery of stalled processes.
Its capability surface covers asynchronous task scheduling for delayed jobs, concurrency control for worker nodes, and job lifecycle management. It includes tools for monitoring queue health, tracking job progress, and retrieving results based on job state.
The system supports bulk job enqueuing to reduce network overhead and allows for custom job identifiers and configurable backoff strategies for failed tasks.
Features
- Redis-Backed Queues - Provides a distributed task queue that uses Redis as the primary persistent storage for jobs.
- Delayed Task Scheduling - Enables queuing functions to run after a specific delay or at a future timestamp.
- Distributed Processing - Distributes computational tasks across multiple worker processes to handle high volumes concurrently.
- Job Concurrency Controllers - Executes jobs using configurable concurrency limits across multiple worker nodes to maximize throughput.
- Job Scheduling - Provides tools for scheduling and executing background jobs at specific future times.
- Stalled Job Recovery - Detects and recovers stalled jobs that stopped making progress due to worker process crashes.
- Polling-Based Worker Execution - Implements a worker execution model where background processes poll the store for pending tasks.
- Node.js Background Workers - Offloads heavy tasks from the Node.js main execution thread to background workers.
- Job Queues - Creates and stores tasks in a persistent store for asynchronous processing.
- Redis-Backed Background Job Libraries - Provides a library for enqueueing and processing asynchronous tasks backed by Redis.
- Reliable Task Queues - Guarantees task completion through automatic retries and the recovery of stalled processes.
- Retry Strategies - Implements configurable retry attempts and backoff strategies, including exponential logic, for failed tasks.
- Lua Scripting - Utilizes server-side Lua scripts to ensure atomic state transitions and thread-safe job acquisition in Redis.
- Job State Tracking - Enables looking up and filtering jobs based on their current state, such as waiting, active, or failed.
- Job Tracking by Identifier - Allows assigning custom unique string identifiers to jobs for easier tracking and referencing.
- Queue and Job State Inspections - Provides metadata and state inspections to monitor the current count of jobs in each queue state.
- Per-Job Progress Reporting - Allows workers to send serializable progress updates that are received as events by the job creator.
- Job Output Retrievers - Provides mechanisms to retrieve the output and results of completed background tasks.
- Job Timeout Enforcements - Enforces maximum execution time limits for jobs and marks them as failed if exceeded.
- Priority-based Sorted Sets - Uses Redis sorted sets to manage delayed jobs based on their future execution timestamps.
- Job Lifecycle Management - Provides capabilities to retrieve, remove, and track the lifecycle of background tasks.
- Bulk Enqueueing - Supports adding multiple jobs to the queue in a single network request to improve throughput.
- Graceful Shutdown Mechanisms - Ensures active jobs finish processing before the system shuts down and terminates connections.
- Job Lifecycle - Provides event listeners to track job lifecycle stages including enqueueing, start, completion, and failure.
- Job Status Monitoring - Tracks the real-time execution progress and completion status of asynchronous processing jobs.
- Job Queues - High-performance Redis-backed job queue.
- Task Queues - High-performance Redis-based task queue.
