jrallison/go-workers
Go Workers
Go-workers is a background job processor for Go applications that coordinates asynchronous task execution using Redis queues. It implements the Sidekiq-compatible wire protocol and payload structures, allowing interoperability with existing background processing ecosystems and clients.
The system manages task distribution through atomic list operations on a shared data store to guarantee reliable message delivery and prevent lost tasks during abrupt worker failures. It features configurable concurrency limits to control throughput per queue by dispatching incoming tasks across bounded synchronization channels, alongside automatic failure retry polling that catches runtime panics and schedules failed messages back into persistence layers according to backoff policies.
Execution lifecycles are customizable through middleware-driven pipelines that intercept job handling events using nested function wrappers. Operational visibility and system lifecycle management are supported via an embedded HTTP endpoint exposing live statistics, alongside signal-based graceful shutdowns that drain active queues and complete running workloads before process termination.
Features
- Redis-Backed Queues - Coordinates background task distribution by reading and writing job payloads through atomic list operations on a shared data store.
- Redis-Backed Background Job Libraries - Executes asynchronous background tasks concurrently using Redis as the persistence store and message broker.
- Automated Job Retries - Automatically catches failed task executions and schedules subsequent retries according to predefined error handling policies.
- Background Job Processing - Executing asynchronous tasks concurrently with adjustable worker limits per queue to offload heavy application workloads.
- Reliable Message Delivery - Guarantees message delivery and prevents lost tasks during abrupt worker failures by using atomic list operations.
- Background Task Processing - Executes asynchronous tasks concurrently with adjustable worker limits per queue to offload heavy application workloads.
- Background Job Processors - A task processing framework for Go applications that supports concurrency limits, retries, and custom middleware.
- Composable Middleware Pipelines - Intercepts job handling lifecycle events through nested function wrappers that execute custom cross-cutting logic before and after task completion.
- Concurrency Limiting - Controls worker throughput per queue by dispatching incoming tasks across bounded synchronization channels to optimize resource utilization.
- Sidekiq-Compatible Protocols - Implements the Sidekiq-compatible wire protocol and payload structures to allow full interoperability with existing ecosystem clients.
- Automatic Task Retries - Catches runtime panics during task execution and schedules failed messages back into persistence layers according to predefined backoff policies.
- Graceful Shutdowns - Listens for operating system signals to pause new work and safely finish active jobs before terminating.
- Instance Graceful Shutdowns - Listens for operating system interrupt signals to drain active queues and safely complete running workloads prior to process termination.
- Worker-Level Concurrency Control - Allows adjusting worker concurrency levels independently for each queue to optimize resource usage and throughput.
- Job Middleware - Applies custom middleware around job processing to run custom logic before and after each task handles a message.
- Wire-Compatible Handlers - Parses and generates JSON payload structures matching established ecosystem conventions to allow interoperability with existing background processing clients.
- Background Job Dashboards - Provides an embedded HTTP server delivering live metrics and status information about currently running background jobs.
- Background Job Monitoring - Exposes operational metrics detailing current worker activity and job statuses for complete operational visibility.
- Server Metrics - Exposes live operational statistics and internal worker health metrics over an embedded web endpoint for external monitoring systems.
Star history
How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
