Bogdanp/dramatiq

Features

• Asynchronous Task Processing - Offloads long-running functions to background workers via Redis or RabbitMQ to keep main applications responsive.
• Distributed Task Workers - Distributes computational workloads across multiple worker processes to handle high volumes of concurrent tasks.
• Message Brokers - Functions as a distributed task queue using RabbitMQ or Redis to decouple asynchronous communication between services.
• Distributed Task Queues - Functions as a distributed task queue that offloads function execution to background workers via RabbitMQ or Redis.
• Output Chaining - Links multiple tasks together in a pipeline so the output of one task is passed as input to the next.
• Task Orchestrators - Links background jobs into sequences or pipelines where the output of one task triggers the start of the next.
• Distributed Task Coordination - Uses shared locks and barriers to coordinate task execution and prevent resource exhaustion across a cluster.
• Distributed Locks - Utilizes shared locks to coordinate access and prevent simultaneous execution of the same task across a cluster.
• Message Queue Workers - Provides background processing components that consume and execute tasks from declared message queues.
• Reliable Message Delivery - Ensures background jobs are delivered and completed through retries, dead letter queues, and persistence mechanisms.
• Distributed Rate Limiting - Uses distributed rate limiters to restrict the frequency and concurrency of background tasks across a cluster.
• Task Queues - Allows declaration of named queues and provides mechanisms to enqueue messages or flush pending tasks.
• Background Process Offloading - Offloads long-running computations to separate background processes to keep the main application responsive.
• Broker Disk Persistence - Ensures reliable message delivery by storing tasks on disk and redelivering them if processing fails.
• Message Delivery Guarantees - Guarantees reliable message delivery between producers and consumers using explicit acknowledgments.
• Exponential Backoff Retries - Re-enqueues failed messages using exponential backoff with configurable limits and predicates.
• Worker Process Consumers - Spawns independent OS processes and threads as consumers that pull tasks from a shared queue for parallel execution.
• Asynchronous Task Orchestrators - Provides a framework for linking background jobs into sequences, pipelines, and barriers to orchestrate complex asynchronous workflows.
• Concurrent Task Limiters - Constrains the number of simultaneous task instances using distributed mutexes to prevent system resource exhaustion.
• Distributed Job Locking - Prevents multiple workers from processing the same job simultaneously using a distributed lock mechanism.
• Task Actors - Provides a mechanism to wrap functions as actors for asynchronous execution via a message broker.
• Backoff Strategies - Implements algorithmic delays between retry attempts that increase exponentially to prevent system overload.
• Background Job Schedulers - Defers the execution of specific tasks to a future time or triggers them based on external events.
• In-Memory Testing - Provides a local in-memory implementation for verifying task behavior without requiring an external network transport.
• Task Result Storage - Saves the return values of background tasks to a remote store for later retrieval by other processes.
• Workflow Result Retrieval - Provides mechanisms to persist and fetch the return values of completed background tasks using unique identifiers.
• Task Completion Callbacks - Executes specific actors automatically when another task succeeds or fails to handle post-processing.
• Task Interruption Handlers - Safely halts active worker threads via exceptions during shutdowns or timeout events.
• Distributed Concurrency Controls - Implements distributed mutexes and rate limiters to prevent resource exhaustion across a cluster of workers.
• Task Execution Prioritization - Assigns importance levels to tasks to ensure high-priority work is processed before lower-priority items.
• Delayed Task Scheduling - Defers the execution of a task to a specific time in the future or after a specified delay.
• Running Task Terminations - Allows stopping active task executions mid-process using unique message identifiers to recover resources.
• Barrier Synchronization - Coordinates multiple remote workers by blocking execution until a specific number of tasks reach a checkpoint.
• Dead Letter Queues - Isolates unprocessable messages that exceed retry or age limits into specialized queues for inspection.
• Queue-Based Worker Routing - Binds specific worker groups to designated queues to control throughput and prioritize different types of work.
• Broker Adapters - Implements plugin-based interfaces that allow different message broker technologies to be swapped interchangeably.
• Broker Integrations - Provides a pluggable transport layer supporting multiple brokers like Redis and RabbitMQ for task payload delivery.
• Async Task Timeouts - Enforces execution time limits on asynchronous operations to prevent workers from hanging.
• Pluggable Transport Backends - Provides a pluggable backend interface to switch between transport protocols like RabbitMQ and Redis without changing application code.
• Concurrent Task Execution - Supports running concurrent workers using lightweight threads like Gevent to increase performance for I/O workloads.
• Distributed Barrier Synchronization - Implements distributed barriers to coordinate multiple remote workers by blocking execution until a specific task count is reached.
• Module Functionality Extenders - Allows implementation of custom middleware classes to hook into the message lifecycle and modify global behavior.
• Middleware Lifecycle Hooks - Provides a middleware system to intercept message enqueueing and acknowledgement for custom global behavior modifications.
• Task Lifecycle Interceptions - Provides middleware hooks to modify job payloads before enqueuing and wrap execution logic for global behavior control.
• In-Memory Broker Stubs - Replaces the remote message broker with a local in-memory implementation for instant synchronous unit test execution.
• Grouped Parallel Execution - Enqueues groups of tasks for concurrent execution and triggers a callback upon collective completion.
• Task Queues - Listed in the “Task Queues” section of the Awesome Python awesome list.
• Task Queues and Background Jobs - Integration for background task processing.
• Task Queues - Reliable background task processing for modern applications.

Star history