resque/resque
Resque
Resque is a Ruby library for enqueueing and processing asynchronous tasks using Redis as a data store. It functions as a distributed task processor and queue manager, allowing long-running work to be moved out of the main request cycle.
The system executes background jobs in isolated child processes to prevent memory leaks and provides a web-based dashboard for monitoring queue depths, worker activity, and failed job statistics.
Capability areas include distributed worker coordination via signals, error handling with job retry mechanisms, and priority-ordered queue management. It also supports lifecycle hooks to extend job behavior and uses namespaces to isolate data keyspaces within Redis.
Features
- Asynchronous Task Processing - Offloads long-running work to background workers using Redis as the queueing data store.
- Distributed Task Processors - Functions as a distributed task processor that executes background operations across multiple worker nodes.
- Redis-Backed Queues - Utilizes Redis as the primary persistent storage and message broker for distributed task queues.
- Background Job Processing - Provides a system for offloading long-running work from the main request cycle to asynchronous background workers.
- Background Job Queues - Persists tasks as data objects into Redis queues to be processed asynchronously by background workers.
- Distributed Task Workers - Coordinates a distributed fleet of workers using Unix signals for lifecycle management and process isolation.
- Process-Based Isolation - Executes every background job in a separate child process to isolate crashes and prevent memory leaks.
- Polling-Based Worker Execution - Implements a polling-based model where workers periodically check Redis queues for new tasks.
- Message Queue Dashboards - Provides a visual dashboard for tracking queue depths and the overall health of the task system.
- Task Queues - Runs background workers that poll Redis queues and execute jobs within isolated child processes.
- Job Process Sandboxing - Spawns a new child process for every job to prevent memory leaks and isolate crashes from the main worker.
- Redis-Backed Background Job Libraries - Provides a Ruby library for enqueueing and processing asynchronous tasks using Redis as a data store.
- Background Job Monitoring - Provides a visual dashboard for monitoring queue depths, worker activity, and job failure statistics.
- Queue Administration Interfaces - Ships a web-based administrative UI for monitoring and managing background job queues.
- Job Monitoring Tools - Offers an administrative interface to monitor the health, activity, and failure rates of background jobs.
- Failed Processing Job Auditing - Captures failed jobs and their exceptions into a dedicated failure queue for auditing and debugging.
- Task Status Monitors - Tracks the operational status of queues, jobs, and workers via a dedicated monitoring interface.
- Namespace Prefixing - Uses namespace prefixing for Redis keys to allow multiple independent environments on a single server.
- Task Schedulers - Implements a task scheduler that manages the execution and lifecycle of persistent background jobs.
- Job Priority Management - Controls the processing order by specifying a sequence of queues for workers to check in a set priority.
- Automatic Retry Mechanisms - Provides mechanisms to re-queue failed jobs individually or in bulk via an interface or API.
- Job Middleware - Implements a system for adding cross-cutting logic like locking via job execution wrappers.
- Shutdown Signaling - Uses Unix signals to manage the lifecycle of background worker processes, including graceful shutdowns.
- Background Task Retry Policies - Provides mechanisms to define retry attempts and backoff intervals for failing background tasks.
- Failure Queues - Provides a dedicated failure queue to capture exceptions and job metadata for later inspection and manual retry.
- Lifecycle Event Hooks - Provides hooks to execute custom logic at predefined stages of the job and worker lifecycles.
- Worker Lifecycle Hooks - Provides predefined trigger points for custom logic to execute before and after job or worker events.
- Priority Queues - Determines job execution order by iterating through a sequence of queues in a specific priority order.
- Background Job Dashboards - Ships a web-based dashboard for monitoring worker activity, queue depths, and failed job statistics.
- Process Monitors - Tracks processing statistics and resource usage for background services via a configurable backend.
- Worker Process Management - Manages background workers as isolated OS-level processes to ensure stability and resource isolation.
- Worker Lifecycle Controls - Provides mechanisms to control worker runtime states, such as graceful shutdowns and pausing, using Unix signals.
- Worker Event Interceptors - Allows executing custom logic at specific worker stages such as before forking or during shutdown.
- Background Jobs - Redis-backed background job queue.
- Queues and Messaging - Redis-backed library for background jobs.
