RichardKnop/machinery

Features

• Asynchronous Task Processing - Provides a distributed system for offloading heavy computations to background workers to keep applications responsive.
• Distributed Task Queues - Provides a distributed task queue for reliable and scalable background job processing across multiple nodes.
• Job Schedulers - Automates recurring or delayed operations across multiple server instances without manual intervention.
• Parallel Task Orchestrators - Runs independent tasks in parallel across remote infrastructure and blocks until the group has finished.
• Sequential Task Dependencies - Executes tasks in a strict sequence where subsequent steps are triggered by the output of previous tasks.
• Task Schedulers - Registers and executes persistent background tasks at specific future times or scheduled intervals.
• Background Job Queues - Manages worker concurrency and utilizes routing to distribute tasks across dedicated queues for resource optimization.
• Distributed Task Workers - Implements distributed workers that execute heavy workloads across multiple processes to avoid blocking the main application.
• Delayed Executions - Schedules background tasks for future processing by associating a specific execution timestamp with the job.
• Message-Based Process Orchestrations - Coordinates multi-step task chains and parallel groups by passing messages between independent distributed services.
• Message Passing - Uses a message queue to coordinate work between distributed services and ensure reliable task processing.
• Result-Passing Chains - Enables the execution of task sequences where results are passed from one job to the next.
• Asynchronous Workflow Management - Provides a comprehensive framework for managing the full lifecycle of distributed asynchronous task workflows.
• Task Coordinations - Implements synchronization workflows that trigger follow-up tasks after a group of parallel jobs has finished.
• Complex Workflow Coordination - Coordinates dependent execution chains and parallel groups of tasks through defined job signatures.
• Result Persistence Layers - Persists the final state and return values of asynchronous operations in a backend store for later retrieval.
• Workflow Schedulers - Automates repetitive background jobs by running tasks or complex workflows on a recurring timetable.
• Automatic Retry Mechanisms - Automatically re-enqueues failed tasks using configurable attempt limits and a gradual backoff strategy.
• Queue-Based Message Distribution - Uses routing keys to direct specific job types to dedicated queues, balancing workloads across worker groups.
• Worker-Level Concurrency Control - Includes mechanisms for managing worker concurrency to prevent system resource exhaustion under heavy load.
• Task Queues - Supports routing keys to organize background tasks into named queues for managed execution and workload balancing.
• Concurrent Task Limiters - Controls the maximum number of active tasks a single worker handles simultaneously to prevent resource exhaustion.
• Job Result Persistence - Provides mechanisms to capture and store return values and metadata from completed background tasks for later retrieval.
• Backoff Strategies - Implements an algorithmic delay between retry attempts that increases after each consecutive failure.
• Messaging Systems - Asynchronous task queue based on distributed message passing.
• Service Governance and Platforms - Asynchronous task queue for background job processing.
• Task Scheduling and Queues - Distributed task queue.
• Messaging - Listed in the “Messaging” section of the Awesome Go awesome list.
• Messaging Systems - Asynchronous task queue for distributed systems.

Star history