Automattic/kue
Kue
Kue is a Redis-backed job queue library for Node.js that provides a complete system for defining, scheduling, and processing background work. It stores job metadata and state in Redis lists and sorted sets, enabling persistent, in-memory operations with configurable concurrency control and priority-sorted processing. The library includes a RESTful HTTP API for managing jobs and a web-based monitoring dashboard for inspecting job status, progress, and logs.
The system distinguishes itself through its event-driven worker model, where workers listen for job events via Redis pub/sub and process jobs asynchronously using callbacks. It supports delayed job scheduling with automatic promotion to the active queue, priority-based execution ordering, and fault-tolerant processing with configurable retries, timeouts, and recovery of stuck jobs after crashes. Graceful shutdown protocols allow workers to finish current jobs within a configurable timeout, while pause and resume controls provide administrative flexibility.
Kue offers job lifecycle monitoring through event listening for enqueue, start, completion, and failure events, along with per-job progress tracking and logging. The web dashboard provides real-time visibility into queue state, and search indexing enables querying jobs by their data content through REST endpoints. The system includes job cleanup for automatic removal of completed work, stuck job recovery on startup, and support for distributing processing across multiple CPU cores using Node.js Cluster. Security measures protect the web interface and API with TLS and HTTP basic authentication.
Features
- Redis-Backed Background Job Libraries - Runs background job queues backed by Redis for persistence, reliability, and fast in-memory operations.
- Delayed Task Scheduling - Schedules jobs to execute at a future time or after a specified delay without manual intervention.
- Per-Type Concurrency Limits - Implements configurable per-job-type concurrency limits using Redis-backed semaphores.
- Job Priority Management - Manages and processes jobs with configurable priority levels to ensure high-priority tasks execute first.
- Priority-Based Job Schedulers - Executes delayed and prioritized work items with configurable concurrency and retry logic.
- Redis Sorted Set Priority Ordering - Orders jobs by priority using Redis sorted sets for deterministic dequeuing.
- Concurrent Job Schedulers - Runs multiple jobs of the same type in parallel with configurable concurrency limits.
- Node.js Job Schedulers - A Node.js library for defining, queuing, and processing background jobs with lifecycle event hooks.
- Redis-Backed Queues - Stores job metadata and state in Redis lists and sorted sets for persistence and atomic operations.
- Event-Driven Callbacks - Workers listen for job events via Redis pub/sub and process jobs asynchronously using callbacks.
- Fault Tolerance Implementation - Handles job failures with automatic retries, timeouts, and recovery of stuck jobs after crashes.
- Automation Job APIs - Provides a RESTful JSON API for creating, retrieving, deleting, and listing background jobs.
- Background Job Dashboards - Provides a browser UI to inspect job status, progress, logs, and search job data.
- Job Status Monitoring - Tracks job progress, logs messages, and views status through a web dashboard for observability.
- RESTful API Designs - Exposes job management operations through a JSON HTTP interface built on top of the queue core.
- Node.js Cluster Distribution - Distributes job processing across multiple CPU cores using Node.js Cluster.
- Graceful Shutdowns - Signals workers to finish current jobs and stop accepting new work within a configurable timeout.
- Completed Job Cleanups - Removes a job from Redis automatically upon completion to free memory.
- Automatic Retry Mechanisms - Automatically re-attempts jobs after failure with configurable retry counts and delays between attempts.
- Stalled Job Recovery - Re-queues jobs left in an active state after a crash by scanning for stale entries on startup.
- Web Interface Protections - Ships TLS and HTTP basic authentication middleware to protect the web dashboard and REST API.
- Queue Monitoring Dashboards - Ships a browser dashboard for viewing job status, progress, logs, and managing queue operations in real time.
- Job Management APIs - Provides a JSON HTTP API for creating, inspecting, and deleting queued jobs with search and filtering.
- Job Lifecycle - Subscribes to lifecycle events like enqueue, start, complete, or failure on jobs or the queue.
- Open-source projects made by Iranian developers - Listed in the “Open-source projects made by Iranian developers” section of the Github Do Not Ban Us awesome list.
