Open-source libraries and frameworks for managing background job processing using PostgreSQL as the primary storage.
.. image:: https://media.charlesleifer.com/blog/photos/huey3-logo.png
Huey is a task queue for Python that supports multiple storage backends, including PostgreSQL, and provides essential features like retries, scheduling, and concurrency control for background job processing.
Airflow is a platform for programmatically authoring, scheduling, and monitoring complex data pipelines. It functions as a workflow automation engine that manages the lifecycle of recurring business processes by executing code-defined task dependencies. By representing workflows as directed acyclic graphs, the system ensures that task execution order and data flow are explicitly defined and reliably maintained across distributed computing environments. The platform distinguishes itself through a highly modular, provider-based architecture that decouples core orchestration logic from external
Airflow is a robust workflow orchestration platform that uses PostgreSQL as its primary metadata store to manage task scheduling, retries, and concurrency, making it a comprehensive solution for complex job processing.
Machinery is a distributed task queue and asynchronous workflow engine. It provides a system for processing heavy workloads outside the main request flow using a network of distributed background workers and a message-based job orchestrator. The project manages complex task lifecycles through sequential chaining, where results are passed between tasks, and parallel coordination, which can trigger callback tasks upon the completion of a group. It supports periodic workflow scheduling for recurring jobs and delayed execution via specific timestamps. The system includes capabilities for result
This is a distributed task queue that relies on message brokers like Redis or RabbitMQ rather than PostgreSQL, making it a different architectural approach than the database-backed solution you requested.
Bull-board is a web-based administrative interface and dashboard for visualizing, monitoring, and managing asynchronous job queues in Node.js. It serves as a control panel for background job administrators to oversee queue processing and manage the lifecycle of tasks powered by BullMQ, Bull, and Redis. The project distinguishes itself by providing a pluggable interface that integrates with various web server frameworks as middleware. It utilizes an adapter-based system to provide unified administrative controls across different queue backends and includes authentication guards to secure the d
This is a monitoring and management dashboard for existing job queues rather than a task queue library itself, meaning it requires a separate backend like BullMQ to function.
Bull is a Node.js library for managing distributed jobs and message queues using Redis as the primary data store. It functions as a distributed task worker, job scheduler, and priority queue manager designed to handle asynchronous workloads across multiple processes. The project distinguishes itself by providing a persistent communication channel that decouples servers through the exchange of serializable data objects. It ensures distributed system reliability by detecting stalled tasks and recovering from process crashes to ensure every queued job is completed. The system covers a broad ran
This is a robust task queue and job processing library, but it is built specifically for Redis rather than PostgreSQL, making it a different architectural choice for your requirements.
Hatchet is an open-source durable workflow engine and task orchestration platform. It provides a framework for building and executing fault-tolerant, multi-step pipelines as directed acyclic graphs (DAGs), with automatic retries, scheduling, and real-time observability. The system is built around durable task checkpointing, which persists execution state after each step so work can resume from the last checkpoint after a worker crash or restart, and it supports event-driven task resumption that pauses a task until a matching external event arrives. The platform distinguishes itself through it
Hatchet is a durable workflow engine that uses PostgreSQL as its primary state store, providing the transactional integrity, scheduling, and observability required for robust task orchestration.
Resque is a Ruby library for enqueueing and processing asynchronous tasks using Redis as a data store. It functions as a distributed task processor and queue manager, allowing long-running work to be moved out of the main request cycle. The system executes background jobs in isolated child processes to prevent memory leaks and provides a web-based dashboard for monitoring queue depths, worker activity, and failed job statistics. Capability areas include distributed worker coordination via signals, error handling with job retry mechanisms, and priority-ordered queue management. It also suppor
This is a task queue and job processing library, but it relies on Redis rather than PostgreSQL for its storage engine and transactional integrity.
Prefect is a workflow orchestration platform designed to define, schedule, and monitor complex data pipelines as Python code. It functions as a container-native engine that wraps individual tasks in isolated environments, ensuring consistent dependencies and resource allocation across diverse infrastructure. By utilizing a state-machine-based orchestration model, the system tracks execution progress through discrete transitions and persistent event logs to maintain reliable and observable task processing. The platform distinguishes itself through a decoupled worker-API architecture, which sep
Prefect is a robust workflow orchestration platform that uses PostgreSQL as its primary backend to manage task state, scheduling, and retries, making it a powerful choice for complex job processing requirements.
Mikro-ORM is a TypeScript-based object-relational mapping system that provides a unified persistence layer for Node.js applications. It translates TypeScript entities into relational or document-based database schemas, supporting a variety of engines including PostgreSQL, MySQL, MariaDB, MS SQL Server, SQLite, and MongoDB. The project implements the data mapper pattern to decouple in-memory domain models from the database persistence layer. It utilizes a unit of work pattern to track entity changes in memory and commit them in a single coordinated database transaction. The library covers com
This is an object-relational mapping library for database interaction rather than a task queue or job processing system, meaning it lacks the scheduling and concurrency features required for managing background jobs.
Convex is a serverless backend platform that provides a real-time reactive database, serverless functions, and state synchronization for web applications. It manages relational JSON documents using ACID-compliant transactions and schema validation to ensure data consistency and integrity. The platform distinguishes itself by synchronizing database state with clients via WebSockets, allowing user interfaces to update automatically as data changes. It also includes a specialized vector search database for performing semantic search using embeddings and supports both cloud-native deployment and
Convex is a comprehensive serverless backend platform that includes task scheduling as a feature, but it is not a dedicated PostgreSQL-backed task queue library designed for transactional job processing.
rq is a distributed task queue and background worker system for Python that uses a Redis backend to decouple task submission from execution. It functions as a reliable message queue and task scheduler, allowing Python functions or asyncio coroutines to be processed asynchronously across multiple worker processes. The project distinguishes itself through reliable queuing mechanisms that prevent job loss during worker crashes using atomic operations. It provides specialized orchestration capabilities, including the prevention of duplicate jobs, job execution prioritization, and the ability to m
This is a task queue system, but it is built specifically for Redis rather than PostgreSQL, making it a different architectural choice than the database-backed solution you requested.
Celery is an asynchronous job processor and distributed task queue designed to offload time-consuming operations to background worker nodes. By utilizing a message-passing architecture, it decouples task producers from consumers, allowing applications to maintain responsiveness while scaling workloads across multiple isolated environments. The system functions as a distributed workload orchestrator that manages the lifecycle of deferred operations through persistent queues. It distinguishes itself by providing a pluggable transport abstraction, which allows the core task logic to remain indep
Celery is a robust distributed task queue that supports PostgreSQL as a result backend and message broker, providing the scheduling, retry, and concurrency features required for transactional job processing.