contribsys/faktory

Features

• Background Job Dispatchers - An open-source work server that queues, dispatches, and manages background jobs across multiple languages.
• Background Job Processing - Manages and executes background tasks asynchronously across multiple languages with queuing and retries.
• Background Job Queues - Provides language-specific client APIs to push background jobs to a work server for asynchronous processing.
• Language-Agnostic Queues - Stores and distributes JSON job payloads to workers written in any language via language-specific APIs.
• Language-Specific Job Fetchers - Ships worker libraries in multiple languages that fetch and execute queued background jobs from the server.
• Automatic Retry Mechanisms - Automatically retries failed jobs with exponential backoff to ensure eventual completion.
• Batch Workflow Trackers - Groups jobs into batches and tracks their completion as a single asynchronous workflow with progress reporting.
• Redis-Backed Queues - Stores job payloads as JSON hashes in Redis queues for reliable persistence and distribution across workers.
• Batch Workflow Orchestrations - Groups jobs into batches and tracks their completion as a single asynchronous workflow across languages.
• Language-Agnostic APIs - Uses a custom TCP protocol with JSON serialization to enable job push and fetch from any programming language.
• Background Job Dashboards - Ships a built-in web dashboard for inspecting queues, retrying jobs, and monitoring worker activity in real time.
• Job Schedulers - Enqueues jobs on a fixed timetable without manual triggering, supporting cron-like schedules for periodic execution.
• Distributed Job Execution - Stores jobs as JSON hashes in queues and distributes them to workers across one or hundreds of machines for execution.
• Job Deduplication - Implements a configurable job uniqueness filter that prevents duplicate enqueues within a time window using Redis.
• Kubernetes Deployment - Runs a single-replica job server inside Kubernetes with persistent storage and ConfigMap configuration.
• Persistent Server Deployments - Runs a persistent background job server inside Kubernetes with configuration, storage, and health probes.
• Worker Throttling - Limits queue processing throughput per time window to prevent worker overload or SLA violations.
• Workflow Batchers - Groups jobs into batches and orchestrates their execution as a single asynchronous workflow with completion tracking.
• Workflow Orchestrators - Includes a batch workflow orchestrator that groups jobs into batches with completion tracking for multi-step workflows.
• Queue Administration Interfaces - Provides a web dashboard to inspect queues, retry jobs, and monitor worker activity in real time.
• StatsD Exporters - Streams job throughput and queue metrics to StatsD for live monitoring and alerting.
• Retry and Backoff Logic - Features an exponential backoff retry engine that automatically requeues failed jobs to handle transient errors.
• Job Status Monitoring - Reports the completion status of individual jobs within a batch for workflow visibility.
• Distributed Messaging - Background job processing system for application tasks.
• Job Scheduling and Automation - Language-agnostic server for persistent background job processing.

Star history