coleifer/huey
Huey
Features
- Background Task Integrations - Integrates Django with a Huey-based background worker for asynchronous task execution.
- Atomic Task Locks - Provides named locks to prevent concurrent execution of the same task.
- Multi-Backend Task Persistence - Persists task queues in Redis, SQLite, or in-memory storage for flexible deployment.
- Task Queue Storage Backends - Lets users select among Redis, SQLite, filesystem, or in-memory storage for the task queue.
- Background Task Schedulers - Runs Python functions asynchronously in a separate process, freeing the main application.
- Job Schedulers - Executes Python functions on a fixed timetable or cron schedule without manual triggering.
- Parallel Callback Aggregation - Runs parallel tasks then invokes a callback with all results in order.
- Map-Reduce Fan-Outs - Fans out parallel tasks and reduces their results with a map/reduce pattern.
- Output Chaining - Chains tasks by piping output from one to the next in sequence.
- Delayed Executions - Delays task execution to a later time or after a specified duration.
- Future-Time Executions - Queues tasks for future execution at a specific time or after a delay.
- Sequential Pipelines - Enqueues sequential pipelines where each task receives the prior result.
- Task Execution Prioritization - Assigns numeric priorities so higher-priority tasks execute first.
- Task Scheduling - Runs functions automatically on a crontab-like schedule.
- Background Task Runners - Enqueues function calls to run asynchronously and returns a handle to retrieve results later.
- Asynchronous Task Processing - Enqueues tasks to run asynchronously in a separate worker process, freeing the main application.
- Task Schedulers - Executes tasks at fixed intervals or specific times via an integrated scheduler.
- Delayed Task Scheduling - Queues functions to run after a delay or at an absolute future time.
- Task Throughput Rate Limiters - Restricts task invocation frequency within a time window with automatic retry.
- Priority Queues - Controls task execution order with priority levels and restricts invocation frequency within time windows.
- Django Task Queue Adapters - Provides a Django integration layer that runs background tasks using the Huey consumer.
- Multi-Backend Implementations - Supports Redis, SQLite, filesystem, and in-memory storage backends for flexible task queuing.
- Task Expirations - Prevents stale tasks from executing after a configurable timeout.
- Asynchronous Task Retries - Automatically re-executes failed tasks with configurable retry limits and delays.
- Result-Passing Chains - Passes the output of one background task as input to the next in a sequence.
- Background Processing - Offloads work to a separate worker process so the main application remains responsive.
- Django Decorator Definitions - Creates tasks and periodic tasks using decorators imported from the Django integration module.
- Redis-Backed Queues - Uses Redis for queue storage, scheduling, and result tracking.
- Task Retry Policies - Automatically re-executes failed tasks with configurable retry limits and delays.
- Grouped Parallel Execution - Enqueues parallel task groups and collects results as a list.
- Task Schedulers - Runs tasks on fixed intervals, cron schedules, or specific times without manual triggering.
- Crontab-Based Schedulers - Ships a crontab-based scheduler for executing periodic tasks on a fixed timetable.
- Worker Process Executions - Starts a worker process that picks up and executes queued tasks with multiple concurrency models.
- In-Memory Queues - Queues tasks in application memory without requiring an external broker for lightweight processing.
- SQLite-Backed Task Queues - Stores tasks in a SQLite database table for polling-based dispatch to consumer processes.
- Greenlet-Based Consumers - Provides a greenlet-based consumer model for concurrent task execution within a single thread.
- Synchronous Task Execution Modes - Executes tasks synchronously like regular function calls when debug mode is enabled.
- Parallel Task Orchestrators - Runs multiple tasks concurrently and collects their results with support for map/reduce patterns.
- Filesystem-Backed Queues - Implements a filesystem-backed task queue for lightweight background processing without external brokers.
- Management Command Consumers - Starts the task consumer using a management command that auto-discovers task modules.
- Async Task Timeouts - Interrupts tasks that run longer than a given number of seconds, raising a timeout error.
- Thread-Based Consumers - Executes tasks concurrently within a single process using threads for lightweight parallelism.
- Worker Process Consumers - Runs parallel worker processes that pull tasks from a shared queue for concurrent execution.
- Immediate Task Executions - Executes tasks synchronously in-process without a separate consumer for testing and development.
- Single Task Revocations - Revokes pending task instances or all instances of a task function, and restores them if not yet consumed.
- Data Pipelines - Lightweight task queue implementation for Python.
- Infrastructure and Tasks - Lightweight multi-threaded task queue.
- Task Queues - Multi-threaded task queue.
- Task Queues and Background Jobs - Lightweight task queue for Python.
- Workflow Orchestration - Task queue.
- Python Packages - Lightweight task queue.
- Task Queues - Lightweight multi-threaded task queue.
