bee-queue/bee-queue
Bee Queue
Bee-queue ist ein Node.js-Hintergrundverarbeitungssystem, das Redis für Job-Queueing und Persistenz verwendet. Es wurde entwickelt, um rechenintensive Aufgaben vom Haupt-Execution-Thread auf Hintergrund-Worker auszulagern und so die Reaktionsfähigkeit der Anwendung aufrechtzuerhalten.
Das Projekt bietet verteilte Job-Verarbeitung, die es Worker-Nodes ermöglicht, über mehrere Prozesse hinweg zu laufen, um große Mengen an Aufgaben gleichzeitig zu bewältigen. Es stellt eine zuverlässige Aufgabenausführung durch automatische Retries und die Wiederherstellung hängengebliebener Prozesse sicher.
Der Funktionsumfang umfasst asynchrones Task-Scheduling für verzögerte Jobs, Concurrency-Control für Worker-Nodes und Job-Lifecycle-Management. Es enthält Tools zur Überwachung des Queue-Status, zur Verfolgung des Job-Fortschritts und zum Abrufen von Ergebnissen basierend auf dem Job-Zustand.
Das System unterstützt Bulk-Job-Enqueuing zur Reduzierung des Netzwerk-Overheads und ermöglicht benutzerdefinierte Job-Identifikatoren sowie konfigurierbare Backoff-Strategien für fehlgeschlagene Aufgaben.
Features
- Redis-Backed Queues - Provides a distributed task queue that uses Redis as the primary persistent storage for jobs.
- Delayed Task Scheduling - Enables queuing functions to run after a specific delay or at a future timestamp.
- Distributed Processing - Distributes computational tasks across multiple worker processes to handle high volumes concurrently.
- Job Concurrency Controllers - Executes jobs using configurable concurrency limits across multiple worker nodes to maximize throughput.
- Job Scheduling - Provides tools for scheduling and executing background jobs at specific future times.
- Stalled Job Recovery - Detects and recovers stalled jobs that stopped making progress due to worker process crashes.
- Polling-Based Worker Execution - Implements a worker execution model where background processes poll the store for pending tasks.
- Node.js Background Workers - Offloads heavy tasks from the Node.js main execution thread to background workers.
- Job Queues - Creates and stores tasks in a persistent store for asynchronous processing.
- Redis-Backed Background Job Libraries - Provides a library for enqueueing and processing asynchronous tasks backed by Redis.
- Reliable Task Queues - Guarantees task completion through automatic retries and the recovery of stalled processes.
- Retry Strategies - Implements configurable retry attempts and backoff strategies, including exponential logic, for failed tasks.
- Lua Scripting - Utilizes server-side Lua scripts to ensure atomic state transitions and thread-safe job acquisition in Redis.
- Job State Tracking - Enables looking up and filtering jobs based on their current state, such as waiting, active, or failed.
- Job Tracking by Identifier - Allows assigning custom unique string identifiers to jobs for easier tracking and referencing.
- Queue and Job State Inspections - Provides metadata and state inspections to monitor the current count of jobs in each queue state.
- Per-Job Progress Reporting - Allows workers to send serializable progress updates that are received as events by the job creator.
- Job Output Retrievers - Provides mechanisms to retrieve the output and results of completed background tasks.
- Job Timeout Enforcements - Enforces maximum execution time limits for jobs and marks them as failed if exceeded.
- Priority-based Sorted Sets - Uses Redis sorted sets to manage delayed jobs based on their future execution timestamps.
- Job Lifecycle Management - Provides capabilities to retrieve, remove, and track the lifecycle of background tasks.
- Bulk Enqueueing - Supports adding multiple jobs to the queue in a single network request to improve throughput.
- Graceful Shutdown Mechanisms - Ensures active jobs finish processing before the system shuts down and terminates connections.
- Job Lifecycle - Provides event listeners to track job lifecycle stages including enqueueing, start, completion, and failure.
- Job Status Monitoring - Tracks the real-time execution progress and completion status of asynchronous processing jobs.
- Job Queues - High-performance Redis-backed job queue.
- Task Queues - High-performance Redis-based task queue.
