contribsys/faktory
Faktory
Faktory is an open-source work server that queues, dispatches, and manages background jobs across multiple programming languages. It stores job payloads as JSON hashes in a Redis-backed queue and provides language-specific client and worker libraries that enable any language to push jobs to the server or fetch and execute them.
The server includes a batch workflow orchestrator that groups jobs into batches with completion tracking for coordinating multi-step asynchronous workflows. It features a configurable job uniqueness filter that prevents duplicate enqueues within a time window, an exponential backoff retry engine that automatically requeues failed jobs, and a recurring job scheduler that enqueues jobs on a fixed timetable. A web dashboard provides a browser-based interface to inspect queues, retry jobs, and monitor worker activity in real time, while StatsD metrics streaming emits real-time job throughput and queue metrics for external monitoring.
The system supports job expiration to remove stale work, queue throttling to limit processing throughput per time window, and hot-reload configuration for live updates. It can connect to an external Redis instance for centralized storage and offers deployment options for Docker, Kubernetes, and AWS ECS, including health probes for container monitoring and CloudWatch integration for metric collection.
Features
- Background Job Dispatchers - An open-source work server that queues, dispatches, and manages background jobs across multiple languages.
- Background Job Processing - Manages and executes background tasks asynchronously across multiple languages with queuing and retries.
- Background Job Queues - Provides language-specific client APIs to push background jobs to a work server for asynchronous processing.
- Language-Agnostic Queues - Stores and distributes JSON job payloads to workers written in any language via language-specific APIs.
- Language-Specific Job Fetchers - Ships worker libraries in multiple languages that fetch and execute queued background jobs from the server.
- Automatic Retry Mechanisms - Automatically retries failed jobs with exponential backoff to ensure eventual completion.
- Batch Workflow Trackers - Groups jobs into batches and tracks their completion as a single asynchronous workflow with progress reporting.
- Redis-Backed Queues - Stores job payloads as JSON hashes in Redis queues for reliable persistence and distribution across workers.
- Batch Workflow Orchestrations - Groups jobs into batches and tracks their completion as a single asynchronous workflow across languages.
- Language-Agnostic APIs - Uses a custom TCP protocol with JSON serialization to enable job push and fetch from any programming language.
- Background Job Dashboards - Ships a built-in web dashboard for inspecting queues, retrying jobs, and monitoring worker activity in real time.
- Job Schedulers - Enqueues jobs on a fixed timetable without manual triggering, supporting cron-like schedules for periodic execution.
- Distributed Job Execution - Stores jobs as JSON hashes in queues and distributes them to workers across one or hundreds of machines for execution.
- Job Deduplication - Implements a configurable job uniqueness filter that prevents duplicate enqueues within a time window using Redis.
- Kubernetes Deployment - Runs a single-replica job server inside Kubernetes with persistent storage and ConfigMap configuration.
- Persistent Server Deployments - Runs a persistent background job server inside Kubernetes with configuration, storage, and health probes.
- Worker Throttling - Limits queue processing throughput per time window to prevent worker overload or SLA violations.
- Workflow Batchers - Groups jobs into batches and orchestrates their execution as a single asynchronous workflow with completion tracking.
- Workflow Orchestrators - Includes a batch workflow orchestrator that groups jobs into batches with completion tracking for multi-step workflows.
- Queue Administration Interfaces - Provides a web dashboard to inspect queues, retry jobs, and monitor worker activity in real time.
- StatsD Exporters - Streams job throughput and queue metrics to StatsD for live monitoring and alerting.
- Retry and Backoff Logic - Features an exponential backoff retry engine that automatically requeues failed jobs to handle transient errors.
- Job Status Monitoring - Reports the completion status of individual jobs within a batch for workflow visibility.
- Distributed Messaging - Background job processing system for application tasks.
- Job Scheduling and Automation - Language-agnostic server for persistent background job processing.
