RichardKnop/machinery
Machinery
Machinery is a distributed task queue and asynchronous workflow engine. It provides a system for processing heavy workloads outside the main request flow using a network of distributed background workers and a message-based job orchestrator.
The project manages complex task lifecycles through sequential chaining, where results are passed between tasks, and parallel coordination, which can trigger callback tasks upon the completion of a group. It supports periodic workflow scheduling for recurring jobs and delayed execution via specific timestamps.
The system includes capabilities for result persistence, automatic task retries with exponential backoff, and worker concurrency limiting. It also utilizes routing keys to distribute specific job types across dedicated queues.
Features
- Asynchronous Task Processing - Provides a distributed system for offloading heavy computations to background workers to keep applications responsive.
- Distributed Task Queues - Provides a distributed task queue for reliable and scalable background job processing across multiple nodes.
- Job Schedulers - Automates recurring or delayed operations across multiple server instances without manual intervention.
- Parallel Task Orchestrators - Runs independent tasks in parallel across remote infrastructure and blocks until the group has finished.
- Sequential Task Dependencies - Executes tasks in a strict sequence where subsequent steps are triggered by the output of previous tasks.
- Task Schedulers - Registers and executes persistent background tasks at specific future times or scheduled intervals.
- Background Job Queues - Manages worker concurrency and utilizes routing to distribute tasks across dedicated queues for resource optimization.
- Distributed Task Workers - Implements distributed workers that execute heavy workloads across multiple processes to avoid blocking the main application.
- Delayed Executions - Schedules background tasks for future processing by associating a specific execution timestamp with the job.
- Message-Based Process Orchestrations - Coordinates multi-step task chains and parallel groups by passing messages between independent distributed services.
- Message Passing - Uses a message queue to coordinate work between distributed services and ensure reliable task processing.
- Result-Passing Chains - Enables the execution of task sequences where results are passed from one job to the next.
- Asynchronous Workflow Management - Provides a comprehensive framework for managing the full lifecycle of distributed asynchronous task workflows.
- Task Coordinations - Implements synchronization workflows that trigger follow-up tasks after a group of parallel jobs has finished.
- Complex Workflow Coordination - Coordinates dependent execution chains and parallel groups of tasks through defined job signatures.
- Result Persistence Layers - Persists the final state and return values of asynchronous operations in a backend store for later retrieval.
- Workflow Schedulers - Automates repetitive background jobs by running tasks or complex workflows on a recurring timetable.
- Automatic Retry Mechanisms - Automatically re-enqueues failed tasks using configurable attempt limits and a gradual backoff strategy.
- Queue-Based Message Distribution - Uses routing keys to direct specific job types to dedicated queues, balancing workloads across worker groups.
- Worker-Level Concurrency Control - Includes mechanisms for managing worker concurrency to prevent system resource exhaustion under heavy load.
- Task Queues - Supports routing keys to organize background tasks into named queues for managed execution and workload balancing.
- Concurrent Task Limiters - Controls the maximum number of active tasks a single worker handles simultaneously to prevent resource exhaustion.
- Job Result Persistence - Provides mechanisms to capture and store return values and metadata from completed background tasks for later retrieval.
- Backoff Strategies - Implements an algorithmic delay between retry attempts that increases after each consecutive failure.
- Messaging Systems - Asynchronous task queue based on distributed message passing.
- Service Governance and Platforms - Asynchronous task queue for background job processing.
- Task Scheduling and Queues - Distributed task queue.
- Messaging - Listed in the “Messaging” section of the Awesome Go awesome list.
- Messaging Systems - Asynchronous task queue based on distributed messaging.
