bee-queue/bee-queue
Bee Queue
Bee-queue es un sistema de procesamiento en segundo plano para Node.js que utiliza Redis para la puesta en cola de trabajos y persistencia. Está diseñado para descargar tareas pesadas del hilo de ejecución principal a trabajadores en segundo plano para mantener la capacidad de respuesta de la aplicación.
El proyecto proporciona procesamiento de trabajos distribuido, permitiendo que los nodos de trabajo se ejecuten a través de múltiples procesos para manejar grandes volúmenes de tareas de forma concurrente. Asegura una ejecución de tareas confiable mediante reintentos automáticos y la recuperación de procesos bloqueados.
Su superficie de capacidades cubre la programación de tareas asíncronas para trabajos diferidos, control de concurrencia para nodos de trabajo y gestión del ciclo de vida de los trabajos. Incluye herramientas para monitorear la salud de la cola, rastrear el progreso de los trabajos y recuperar resultados basados en el estado del trabajo.
El sistema soporta la puesta en cola de trabajos en bloque para reducir la sobrecarga de red y permite identificadores de trabajo personalizados y estrategias de backoff configurables para tareas fallidas.
Features
- Redis-Backed Queues - Provides a distributed task queue that uses Redis as the primary persistent storage for jobs.
- Delayed Task Scheduling - Enables queuing functions to run after a specific delay or at a future timestamp.
- Distributed Processing - Distributes computational tasks across multiple worker processes to handle high volumes concurrently.
- Job Concurrency Controllers - Executes jobs using configurable concurrency limits across multiple worker nodes to maximize throughput.
- Job Scheduling - Provides tools for scheduling and executing background jobs at specific future times.
- Stalled Job Recovery - Detects and recovers stalled jobs that stopped making progress due to worker process crashes.
- Polling-Based Worker Execution - Implements a worker execution model where background processes poll the store for pending tasks.
- Node.js Background Workers - Offloads heavy tasks from the Node.js main execution thread to background workers.
- Job Queues - Creates and stores tasks in a persistent store for asynchronous processing.
- Redis-Backed Background Job Libraries - Provides a library for enqueueing and processing asynchronous tasks backed by Redis.
- Reliable Task Queues - Guarantees task completion through automatic retries and the recovery of stalled processes.
- Retry Strategies - Implements configurable retry attempts and backoff strategies, including exponential logic, for failed tasks.
- Lua Scripting - Utilizes server-side Lua scripts to ensure atomic state transitions and thread-safe job acquisition in Redis.
- Job State Tracking - Enables looking up and filtering jobs based on their current state, such as waiting, active, or failed.
- Job Tracking by Identifier - Allows assigning custom unique string identifiers to jobs for easier tracking and referencing.
- Queue and Job State Inspections - Provides metadata and state inspections to monitor the current count of jobs in each queue state.
- Per-Job Progress Reporting - Allows workers to send serializable progress updates that are received as events by the job creator.
- Job Output Retrievers - Provides mechanisms to retrieve the output and results of completed background tasks.
- Job Timeout Enforcements - Enforces maximum execution time limits for jobs and marks them as failed if exceeded.
- Priority-based Sorted Sets - Uses Redis sorted sets to manage delayed jobs based on their future execution timestamps.
- Job Lifecycle Management - Provides capabilities to retrieve, remove, and track the lifecycle of background tasks.
- Bulk Enqueueing - Supports adding multiple jobs to the queue in a single network request to improve throughput.
- Graceful Shutdown Mechanisms - Ensures active jobs finish processing before the system shuts down and terminates connections.
- Job Lifecycle - Provides event listeners to track job lifecycle stages including enqueueing, start, completion, and failure.
- Job Status Monitoring - Tracks the real-time execution progress and completion status of asynchronous processing jobs.
- Job Queues - High-performance Redis-backed job queue.
- Task Queues - High-performance Redis-based task queue.
