rq/rq
Rq
rq is a distributed task queue and background worker system for Python that uses a Redis backend to decouple task submission from execution. It functions as a reliable message queue and task scheduler, allowing Python functions or asyncio coroutines to be processed asynchronously across multiple worker processes.
The project distinguishes itself through reliable queuing mechanisms that prevent job loss during worker crashes using atomic operations. It provides specialized orchestration capabilities, including the prevention of duplicate jobs, job execution prioritization, and the ability to manage worker lifecycles via real-time control signals.
The system covers a broad range of automation capabilities, including periodic and recurring job scheduling via cron syntax and the management of complex workflows through job dependency tracking and retry logic. It also supports job status tracking, result capture, and configurable job serialization.
Worker pool orchestration and process control are managed through a command-line interface.
Features
- Distributed Task Queues - Distributes units of work across multiple worker processes using a Redis backend for scalable task processing.
- Redis-Backed Queues - Uses Redis as the primary backend for managing task lists and job metadata through atomic operations.
- Background Task Schedulers - Automates recurring maintenance tasks and periodic reports using cron syntax and future timestamps.
- Asynchronous Task Processing - Offloads heavy Python functions to background workers to maintain web application responsiveness.
- Background Job Processing - Decouples job submission from execution by managing tasks that run asynchronously in the background.
- Background Job Queues - Implements a worker system and Redis store to execute function calls asynchronously.
- Message Queues - Implements a reliable message queue that prevents job loss during worker crashes using atomic operations.
- Queue Worker Management - Includes tools for orchestrating, monitoring, and scaling long-lived processes that consume tasks from Redis.
- Asynchronous Task Queues - Provides a framework for offloading long-running Python processes to background workers via a queue.
- Reliability Patterns - Utilizes the Redis LMOVE pattern to ensure tasks are not lost if a worker fails during execution.
- Reliable Task Queues - Implements atomic Redis list operations to prevent job loss if a worker fails during execution.
- Dependency-Aware Task Orchestration - Manages complex workflows by tracking job dependencies and implementing automatic retry logic.
- Background Job Schedulers - Provides background worker processes that monitor queues and execute pending Python jobs independently.
- Background Processing Workers - Provides a process-based execution environment to run asynchronous Python functions and coroutines.
- Cron Scheduling - Provides a dedicated scheduler process to execute recurring tasks based on cron expressions.
- Dependency Graph Runners - Allows tasks to be executed according to a directed acyclic dependency graph to ensure prerequisites are met.
- Job Schedulers - Automates recurring work by running tasks at fixed intervals or specific future timestamps.
- Process-Based Isolation - Uses fork-based execution to isolate the worker's main loop from job crashes.
- Automatic Retry Mechanisms - Provides mechanisms to re-run tasks that encountered exceptions with custom queue positioning.
- Prerequisite Tracking - Enables prerequisite jobs to be required before a dependent task is allowed to start.
- Worker Pool Management - Provides a command-line interface for managing and scaling worker pool concurrency and processing capacity.
- Job Enqueueing Serialization - Allows the use of custom serializers or JSON to encode job data for storage in Redis.
- Non-blocking IO Architectures - Supports the execution of non-blocking asyncio coroutines within worker processes to handle high-volume I/O.
- Asyncio Coroutine Execution - Supports the execution of non-blocking Python coroutines to handle high-volume I/O workloads.
- Job Monitoring Tools - Tracks and monitors the state transitions of jobs from queued to finished or failed.
- Worker Lifecycle Controls - Implements real-time control signals to pause, resume, and shut down worker processes.
- Worker Process Signaling - Uses a publish-subscribe pattern to send real-time control signals to running background worker processes.
- Data Pipelines - Simple job queue implementation for Python applications.
- Task Queues - Simple job queue system.
