coleiferhuey

0

View on GitHub

5,933 stars392 forksPythonmit1 viewhuey.readthedocs.io

Huey

.. image:: https://media.charlesleifer.com/blog/photos/huey3-logo.png

Features

Background Task Integrations - Integrates Django with a Huey-based background worker for asynchronous task execution.

Atomic Task Locks - Provides named locks to prevent concurrent execution of the same task.

Multi-Backend Task Persistence - Persists task queues in Redis, SQLite, or in-memory storage for flexible deployment.

Task Queue Storage Backends - Lets users select among Redis, SQLite, filesystem, or in-memory storage for the task queue.

Background Task Schedulers - Runs Python functions asynchronously in a separate process, freeing the main application.

Job Schedulers - Executes Python functions on a fixed timetable or cron schedule without manual triggering.

Parallel Callback Aggregation - Runs parallel tasks then invokes a callback with all results in order.

Map-Reduce Fan-Outs - Fans out parallel tasks and reduces their results with a map/reduce pattern.

Output Chaining - Chains tasks by piping output from one to the next in sequence.

Delayed Executions - Delays task execution to a later time or after a specified duration.

Future-Time Executions - Queues tasks for future execution at a specific time or after a delay.

Sequential Pipelines - Enqueues sequential pipelines where each task receives the prior result.

Task Execution Prioritization - Assigns numeric priorities so higher-priority tasks execute first.

Task Scheduling - Runs functions automatically on a crontab-like schedule.

Background Task Runners - Enqueues function calls to run asynchronously and returns a handle to retrieve results later.

Asynchronous Task Processing - Enqueues tasks to run asynchronously in a separate worker process, freeing the main application.

Task Schedulers - Executes tasks at fixed intervals or specific times via an integrated scheduler.

Delayed Task Scheduling - Queues functions to run after a delay or at an absolute future time.

Task Throughput Rate Limiters - Restricts task invocation frequency within a time window with automatic retry.

Priority Queues - Controls task execution order with priority levels and restricts invocation frequency within time windows.

Django Task Queue Adapters - Provides a Django integration layer that runs background tasks using the Huey consumer.

Multi-Backend Implementations - Supports Redis, SQLite, filesystem, and in-memory storage backends for flexible task queuing.

Task Expirations - Prevents stale tasks from executing after a configurable timeout.

Asynchronous Task Retries - Automatically re-executes failed tasks with configurable retry limits and delays.

Result-Passing Chains - Passes the output of one background task as input to the next in a sequence.

Background Processing - Offloads work to a separate worker process so the main application remains responsive.

Django Decorator Definitions - Creates tasks and periodic tasks using decorators imported from the Django integration module.

Redis-Backed Queues - Uses Redis for queue storage, scheduling, and result tracking.

Task Retry Policies - Automatically re-executes failed tasks with configurable retry limits and delays.

Grouped Parallel Execution - Enqueues parallel task groups and collects results as a list.

Task Schedulers - Runs tasks on fixed intervals, cron schedules, or specific times without manual triggering.

Crontab-Based Schedulers - Ships a crontab-based scheduler for executing periodic tasks on a fixed timetable.

Worker Process Executions - Starts a worker process that picks up and executes queued tasks with multiple concurrency models.

In-Memory Queues - Queues tasks in application memory without requiring an external broker for lightweight processing.

SQLite-Backed Task Queues - Stores tasks in a SQLite database table for polling-based dispatch to consumer processes.

Greenlet-Based Consumers - Provides a greenlet-based consumer model for concurrent task execution within a single thread.

Synchronous Task Execution Modes - Executes tasks synchronously like regular function calls when debug mode is enabled.

Parallel Task Orchestrators - Runs multiple tasks concurrently and collects their results with support for map/reduce patterns.

Filesystem-Backed Queues - Implements a filesystem-backed task queue for lightweight background processing without external brokers.

Management Command Consumers - Starts the task consumer using a management command that auto-discovers task modules.

Async Task Timeouts - Interrupts tasks that run longer than a given number of seconds, raising a timeout error.

Thread-Based Consumers - Executes tasks concurrently within a single process using threads for lightweight parallelism.

Worker Process Consumers - Runs parallel worker processes that pull tasks from a shared queue for concurrent execution.

Immediate Task Executions - Executes tasks synchronously in-process without a separate consumer for testing and development.

Single Task Revocations - Revokes pending task instances or all instances of a task function, and restores them if not yet consumed.

Infrastructure and Tasks - Lightweight multi-threaded task queue.

Task Queues - Multi-threaded task queue.

Task Queues and Background Jobs - Lightweight task queue for Python.

Python Packages - Lightweight task queue.

Task Queues - Lightweight multi-threaded task queue.

Huey

Features

Star history