sorentwo/oban
Oban
Oban is a relational database-backed job processor and task scheduler for Elixir applications. It functions as a distributed workflow orchestrator that leverages database primitives to ensure job persistence, reliability, and consistent state management across clusters.
The system distinguishes itself through complex workflow orchestration, enabling the composition of task dependencies using batching, sequential processing, and fan-out or fan-in patterns. It provides global throughput limiting and uniqueness enforcement to prevent duplicate work and protect downstream resources across distributed nodes.
The project covers a broad range of capabilities including scheduled and recurring execution, named queue segmentation, and automated retry logic for failed tasks. It includes observability tools for tracking queue health and job performance metrics via telemetry event streams.
Administrative controls allow for the dynamic configuration of job priorities and the pausing or resuming of queues at runtime.
Features
- Background Job Processing - Provides a distributed system for offloading and managing asynchronous tasks across a cluster using a relational database.
- Relational Database Job Queues - Leverages a relational database as the primary source of truth for job persistence, reliability, and state management.
- Relational Database Processors - Uses a relational database as the single source of truth for task persistence and scheduling, eliminating the need for separate brokers.
- Background Task Schedulers - Manages the execution of delayed and recurring background jobs with integrated retry logic and failure handling.
- Lifecycle State Machines - Manages job transitions through a formal state machine including available, executing, and retryable states to ensure reliable delivery.
- Distributed Concurrency Controls - Limits the number of simultaneous tasks across a distributed cluster to protect downstream resources and prevent system overload.
- Concurrency Control - Limits the number of parallel jobs executing within a specific queue to prevent system overload.
- Distributed Workflow Orchestrators - Coordinates complex sequences of distributed tasks and manages their persistent execution state across multiple nodes.
- Job Queues - Implements named queues to segment workloads and isolate resource consumption across different job categories.
- Recurring Job Scheduling - Enables tasks to be scheduled for execution at a specific future time or on a recurring timetable.
- Elixir Job Queues - Provides a durable, database-backed task processor specifically designed for Elixir applications.
- Job Lifecycle Management - Tracks tasks through a defined set of states, such as available, executing, and retryable, to manage the full job lifecycle.
- Database-Backed Deferred Queues - Uses a relational database as the primary persistent store for scheduling and managing the lifecycle of background tasks.
- Concurrency-Limited Queues - Provides named queues with independent concurrency limits to isolate failures and protect downstream resources from overload.
- Distributed Workflow Orchestration - Coordinates complex task dependencies using batching, sequential processing, and fan-out/fan-in patterns.
- Complex Workflow Coordination - Enables the coordination of complex task dependencies using batching, sequential processing, and fan-out/fan-in patterns.
- Global Job Concurrency Limits - Enforces a maximum number of simultaneous background jobs across a distributed cluster to protect downstream resources.
- Job - Prevents duplicate jobs from being queued based on unique identifiers derived from arguments or queues.
- Database Event Listeners - Utilizes database-level notify/listen events to wake up workers immediately when new jobs are available, reducing polling overhead.
- Job State Querying - Provides capabilities to retrieve and modify background job records using filtered database queries.
- Database-Driven Schedulers - Implements a background job system that uses PostgreSQL primitives for scheduling and state management.
- Runtime Queue Management - Allows starting, stopping, pausing, and resuming queues at runtime without requiring an application restart.
- Automated Job Retries - Automatically reschedules failed tasks for execution with support for custom retry limits and overrides.
- Queue Pause and Resume Controls - Provides controls to temporarily stop a queue from receiving new jobs for maintenance or resource timing.
- Distributed Leader Election - Implements cluster-wide coordination using database advisory locks to ensure only one node acts as the leader for specific operations.
- Job Priority Management - Assigns importance levels to tasks within a queue to ensure high-priority work is executed first.
- Dynamic Job Managers - Allows for the modification of job priorities, schedules, and scaling parameters at runtime without requiring an application restart.
- Job Cancellation - Provides mechanisms to terminate active jobs and trigger necessary resource cleanup across the cluster.
- Transactional Enqueueing - Ensures atomic job scheduling by wrapping the enqueueing process within the same database transaction as the business logic.
- Unique Job Enqueueing - Supports reliable asynchronous task insertion with built-in uniqueness constraints to prevent duplicate work.
- Job Batching - Groups multiple background tasks into batches to track collective progress and trigger completion callbacks.
- Execution Telemetry Pipelines - Streams structured lifecycle events and runtime metrics via telemetry to external monitoring systems for real-time health tracking.
- Queue Performance Metrics - Records runtime metrics for every queue to track cluster performance and system health.
- Job Lifecycle Telemetry - Integrates with telemetry systems to stream structured lifecycle events for logging, error reporting, and health checks.
- Message Queues - Asynchronous job processor powered by PostgreSQL.
- Embedded Workflow Libraries - Robust job processing library for Elixir applications.
