Asynq

Asynq is a distributed background job processing framework for Go applications. It manages asynchronous task queues by offloading heavy operations to persistent storage, allowing the main application to remain responsive while background workers handle workloads.

The system utilizes Redis to manage task state, concurrency, and message distribution across multiple worker instances. It employs atomic Lua scripting and sorted sets to ensure reliable job acquisition, precise scheduling of delayed tasks, and fault-tolerant processing through a two-stage acknowledgement flow.

The framework supports concurrent task execution to scale throughput across distributed environments. It provides capabilities for managing job priorities, retries, and recurring schedules to automate maintenance and background operations.

Features

Distributed Task Queues - Uses Redis to manage task queues, concurrency, and scheduled execution for background job processing.

Job Schedulers - Provides a framework for executing asynchronous tasks across multiple worker instances with support for retries and priorities.

Redis-Backed Queues - Uses a persistent key-value store to manage job state and message distribution across distributed worker processes.

Persistent Background Workers - Offloads heavy operations to background processes to maintain application responsiveness and scale throughput.

Task Queuing Systems - Manages asynchronous work queues using Redis as a persistent storage layer to ensure reliable task delivery.

Background Task Schedulers - Offloads heavy operations to persistent storage for asynchronous processing so the main application remains responsive.

Background Job Processing - Offloads heavy operations to background workers to keep the main application responsive and improve overall system performance.

Messaging Systems - Distributed task queue built on top of Redis.

Task Scheduling and Queues - Distributed task queue using Redis.

Messaging - Listed in the “Messaging” section of the Awesome Go awesome list.

Messaging Systems - Distributed task queue based on Redis.

Job Schedulers - Defines specific timestamps or recurring intervals for job execution to ensure automated maintenance.

Distributed Worker Orchestration - Spawns multiple independent processes that poll the shared data store to execute background tasks in parallel.

Priority-based Sorted Sets - Leverages sorted sets to efficiently track and trigger future tasks based on precise execution timestamps.

Concurrent Task Runners - Scales processing power by running multiple background jobs in parallel across distributed worker instances.

Concurrent Task Limiters - Processes multiple background tasks in parallel across distributed worker instances to increase total throughput.

Reliable Event Delivery Systems - Implements a two-stage processing flow where jobs are moved to an in-progress state until successful completion or failure.

Work Polling - Employs a non-blocking polling mechanism that allows workers to dynamically claim pending jobs from the shared queue.

hibikenasynq

Features

Star history