bee-queue/bee-queue
Bee Queue
Bee-queue 是一个 Node.js 后台处理系统，使用 Redis 进行作业排队和持久化。它旨在将繁重任务从主执行线程卸载到后台 worker，以保持应用响应性。
该项目提供分布式作业处理，允许 worker 节点跨多个进程运行以并发处理大量任务。它通过自动重试和停滞进程的恢复确保可靠的任务执行。
其能力范围涵盖延迟作业的异步任务调度、worker 节点的并发控制以及作业生命周期管理。它包含用于监控队列健康、跟踪作业进度以及根据作业状态检索结果的工具。
该系统支持批量作业入队以减少网络开销，并允许为失败任务设置自定义作业标识符和可配置的退避策略。
Features
- Redis-Backed Queues - Provides a distributed task queue that uses Redis as the primary persistent storage for jobs.
- Delayed Task Scheduling - Enables queuing functions to run after a specific delay or at a future timestamp.
- Distributed Processing - Distributes computational tasks across multiple worker processes to handle high volumes concurrently.
- Job Concurrency Controllers - Executes jobs using configurable concurrency limits across multiple worker nodes to maximize throughput.
- Job Scheduling - Provides tools for scheduling and executing background jobs at specific future times.
- Stalled Job Recovery - Detects and recovers stalled jobs that stopped making progress due to worker process crashes.
- Polling-Based Worker Execution - Implements a worker execution model where background processes poll the store for pending tasks.
- Node.js Background Workers - Offloads heavy tasks from the Node.js main execution thread to background workers.
- Job Queues - Creates and stores tasks in a persistent store for asynchronous processing.
- Redis-Backed Background Job Libraries - Provides a library for enqueueing and processing asynchronous tasks backed by Redis.
- Reliable Task Queues - Guarantees task completion through automatic retries and the recovery of stalled processes.
- Retry Strategies - Implements configurable retry attempts and backoff strategies, including exponential logic, for failed tasks.
- Lua Scripting - Utilizes server-side Lua scripts to ensure atomic state transitions and thread-safe job acquisition in Redis.
- Job State Tracking - Enables looking up and filtering jobs based on their current state, such as waiting, active, or failed.
- Job Tracking by Identifier - Allows assigning custom unique string identifiers to jobs for easier tracking and referencing.
- Queue and Job State Inspections - Provides metadata and state inspections to monitor the current count of jobs in each queue state.
- Per-Job Progress Reporting - Allows workers to send serializable progress updates that are received as events by the job creator.
- Job Output Retrievers - Provides mechanisms to retrieve the output and results of completed background tasks.
- Job Timeout Enforcements - Enforces maximum execution time limits for jobs and marks them as failed if exceeded.
- Priority-based Sorted Sets - Uses Redis sorted sets to manage delayed jobs based on their future execution timestamps.
- Job Lifecycle Management - Provides capabilities to retrieve, remove, and track the lifecycle of background tasks.
- Bulk Enqueueing - Supports adding multiple jobs to the queue in a single network request to improve throughput.
- Graceful Shutdown Mechanisms - Ensures active jobs finish processing before the system shuts down and terminates connections.
- Job Lifecycle - Provides event listeners to track job lifecycle stages including enqueueing, start, completion, and failure.
- Job Status Monitoring - Tracks the real-time execution progress and completion status of asynchronous processing jobs.
- Job Queues - High-performance Redis-backed job queue.
- Task Queues - High-performance Redis-based task queue.
