rq/rq

Features

• Distributed Task Queues - Distributes units of work across multiple worker processes using a Redis backend for scalable task processing.
• Redis-Backed Queues - Uses Redis as the primary backend for managing task lists and job metadata through atomic operations.
• Background Task Schedulers - Automates recurring maintenance tasks and periodic reports using cron syntax and future timestamps.
• Asynchronous Task Processing - Offloads heavy Python functions to background workers to maintain web application responsiveness.
• Background Job Processing - Decouples job submission from execution by managing tasks that run asynchronously in the background.
• Background Job Queues - Implements a worker system and Redis store to execute function calls asynchronously.
• Message Queues - Implements a reliable message queue that prevents job loss during worker crashes using atomic operations.
• Queue Worker Management - Includes tools for orchestrating, monitoring, and scaling long-lived processes that consume tasks from Redis.
• Asynchronous Task Queues - Provides a framework for offloading long-running Python processes to background workers via a queue.
• Reliability Patterns - Utilizes the Redis LMOVE pattern to ensure tasks are not lost if a worker fails during execution.
• Reliable Task Queues - Implements atomic Redis list operations to prevent job loss if a worker fails during execution.
• Dependency-Aware Task Orchestration - Manages complex workflows by tracking job dependencies and implementing automatic retry logic.
• Background Job Schedulers - Provides background worker processes that monitor queues and execute pending Python jobs independently.
• Background Processing Workers - Provides a process-based execution environment to run asynchronous Python functions and coroutines.
• Cron Scheduling - Provides a dedicated scheduler process to execute recurring tasks based on cron expressions.
• Dependency Graph Runners - Allows tasks to be executed according to a directed acyclic dependency graph to ensure prerequisites are met.
• Job Schedulers - Automates recurring work by running tasks at fixed intervals or specific future timestamps.
• Process-Based Isolation - Uses fork-based execution to isolate the worker's main loop from job crashes.
• Automatic Retry Mechanisms - Provides mechanisms to re-run tasks that encountered exceptions with custom queue positioning.
• Prerequisite Tracking - Enables prerequisite jobs to be required before a dependent task is allowed to start.
• Worker Pool Management - Provides a command-line interface for managing and scaling worker pool concurrency and processing capacity.
• Job Enqueueing Serialization - Allows the use of custom serializers or JSON to encode job data for storage in Redis.
• Non-blocking IO Architectures - Supports the execution of non-blocking asyncio coroutines within worker processes to handle high-volume I/O.
• Asyncio Coroutine Execution - Supports the execution of non-blocking Python coroutines to handle high-volume I/O workloads.
• Job Monitoring Tools - Tracks and monitors the state transitions of jobs from queued to finished or failed.
• Worker Lifecycle Controls - Implements real-time control signals to pause, resume, and shut down worker processes.
• Worker Process Signaling - Uses a publish-subscribe pattern to send real-time control signals to running background worker processes.
• Task Queues - Simple job queue system.

Star history