hatchet-dev/hatchet

Features

• Durable Task Orchestrators - Composes multiple tasks into pipelines with dependencies, retries, and checkpointing for reliable execution.
• Directed Acyclic Graph Execution Engines - Declares task dependencies upfront and executes them in topological order with automatic parallelism and state persistence.
• DAG Workflow Executions - Executes workflows as directed acyclic graphs with automatic parallelism and state persistence.
• Durable Workflow Engines - Executes long-running, fault-tolerant workflows that persist state and recover from failures automatically.
• Durable Task Tool Exposures - Converts Hatchet tasks into tool definitions that an AI agent can discover and call through the Model Context Protocol.
• Human-in-the-Loop Workflows - Orchestrates multi-step workflows that pause for human input and resume automatically.
• Durable Multi-Step Orchestrators - Builds durable pipelines of steps with waiting, timeouts, and event-driven transitions.
• Event-Driven Workflow Pauses - Pauses workflows until a scoped event arrives or a deadline passes.
• Durable Event and Timeout Waits - Pauses durable tasks until a sleep duration elapses or an external event arrives, resuming from checkpoints.
• Standalone Task Definitions - Declares standalone tasks that can be run, scheduled, or triggered independently.
• Task Scheduling and Queues - Schedules and runs background tasks on long-running workers with automatic retries and concurrency control.
• Workflow State Stores - Persists all workflow state, task metadata, and event logs in PostgreSQL for durability and observability.
• Parent Output Retrievals - Fetches the result of a parent task in a DAG workflow for use by downstream tasks.
• Parallel Output Merging - Merges results from multiple parallel parent tasks into a single formatted output.
• Workflow Result Retrieval - Returns typed task results synchronously or asynchronously, replacing Celery's result retrieval pattern.
• Background Task Schedulers - Runs one-off or recurring tasks at a future time with configurable retry policies and exponential backoff.
• Cron Scheduling - Defines recurring task schedules directly on task definitions using cron expressions.
• Event-Driven Workflow Triggers - Triggers workflows and tasks from external webhooks, Slack commands, and custom events with real-time processing.
• DAG-Based Orchestration - Defines directed acyclic graphs to model data pipelines and multi-step workflows.
• Parallel Task Orchestrators - Executes multiple tasks concurrently within a workflow by defining them without dependencies.
• Millions-Scale - Handles millions of concurrent task runs with worker-level slot control, fairness, and priority scheduling.
• Task Scheduling - Defines a cron expression in a task definition to automatically enqueue the task on a repeating schedule.
• Webhook Triggers - Ships webhook triggers that receive external HTTP events and initiate automated workflow executions.
• Webhook-Triggered Workflows - Accepts incoming HTTP requests from external services and routes them to appropriate workflows for real-time processing.
• Workflow State Recoveries - Persists full workflow state across worker restarts and resumes execution on any available worker.
• Workflow Schedulers - Creates and manages one-time or recurring scheduled workflow executions.
• Webhook Triggers - Starts workflows automatically when matching webhook events arrive.
• Step-Level Checkpoints - Persists task execution state after each step so work can resume from the last checkpoint after a worker crash or restart.
• Background Job Processing - Queues and executes background tasks with retries, timeouts, concurrency controls, and worker management.
• Distributed Worker Orchestration - Connects workers written in any language over gRPC to a central orchestration engine.
• Named Workers - Provides a programmatic API for creating named workers with configurable slots and labels.
• Control Plane Deployment Tools - Runs the central orchestration system that manages workflows, schedules tasks, and coordinates workers.
• Pause and Resume Strategies - Halts workflow execution on a sleep timer or event condition and resumes when the condition is met.
• Event-Driven Triggers - Starts task execution in response to external events or webhooks for building event-driven systems.
• CEL Expression Filters - Matches incoming event payloads against Common Expression Language rules to determine which waiting task should resume.
• Configurable Task Definitions - Wraps functions as tasks with configurable retries, timeouts, rate limits, and worker affinity.
• Workflow Definitions as Code - Defines tasks and workflows as code with the engine handling durability, retries, and concurrency.
• Sequential Workflows - Defines linear sequences of dependent tasks using parent declarations for execution order.
• Self-Hosted Deployment Platforms - Deploys the full platform on-premise using only PostgreSQL as a dependency.
• Self-Hosted Infrastructure - Operates the full orchestration stack on private infrastructure for full control.
• Self-Hosted Instances - Runs the orchestration engine on user-controlled infrastructure for data and network control.
• Worker Connections - Connects workers to a self-hosted control plane for task execution.
• Task Worker Configurations - Registers long-running worker processes that receive task assignments from the orchestration engine.
• Webhook Triggers - Provides webhook triggers that fire tasks automatically when third-party services send events.
• Connection Configurations - Ships a worker process that must be configured with authentication and server endpoints to connect to the orchestration engine.
• Worker Registration Routing - Routes tasks to workers via registration lists instead of queue-based routing.
• Event Push APIs - Provides APIs to push external events that trigger workflow executions.
• Declarative Workflow Definitions - Declares named workflows with configurable triggers, concurrency, priority, and default task settings.
• Workflow Event Emitters - Emits events into the system to trigger workflows or tasks that listen for them.
• Incoming Webhooks - Creates dedicated HTTP endpoints that external services call to trigger workflows in real time.
• Polyglot Worker Protocols - Connects workers written in any language to the orchestration engine over gRPC for language-agnostic task execution.
• Worker Affinity Rules - Routes tasks to specific workers using labels, affinity rules, and weighted scheduling criteria.
• Webhook Event Consumers - Configures webhook endpoints that accept HTTP requests and route them to tasks by event type.
• Typed Task Invocations - Replaces Celery's .delay() and .applyasync() with .run() or .aiorun() that accept a typed input model and return the result directly.
• Multi-Tenant Identity Management - Supports multiple teams on a single instance with user accounts, roles, and tenant isolation.
• Concurrent Task Limiters - Configures per-worker slot limits to cap concurrent task execution and queue excess work.
• DAG-Based Dependency Resolution - Models workflows as directed acyclic graphs with declared task dependencies and typed data passing.
• Durable Task Conversion - Wraps functions as durable tasks that survive worker restarts with configurable retries and timeouts.
• Durable Execution Pausing - Suspends durable tasks for a duration or until an event occurs without blocking the worker.
• Durable Workflow Execution Engines - Provides an open-source engine for building and executing fault-tolerant, multi-step workflows with automatic retries and scheduling.
• Workflow Dispatching - Triggers workflows with input data and tracks execution through durable state transitions.
• Durable Duration Pausing - Pauses tasks for a specified period without consuming resources, surviving crashes and restarts.
• Durable Time-Based Pausing - Pauses tasks until a specified absolute time, surviving worker crashes and restarts.
• Orchestration Platforms - Defines, schedules, and runs background tasks with configurable retries, timeouts, concurrency controls, and event-driven triggers.
• Retry Policies - Sets retry count, backoff factor, and maximum backoff on tasks, with support for non-retryable exceptions.
• Dynamic Task Spawning - Creates fan-out child task runs from within a parent when the number of parallel items is only known at runtime.
• Task Result Aggregation - Gathers outputs of multiple parallel tasks into a single unified structure after all complete.
• Task Retry Policies - Sets independent retry policies, timeouts, and concurrency limits for each step in a workflow.
• Typed Data Passing - Provides typed data passing between dependent tasks in a DAG workflow.
• Parallel Task Executors - Declares tasks with no dependencies so they execute concurrently within a workflow.
• Immediate Task Executions - Executes workflows or tasks immediately with optional synchronous result retrieval.
• Monitoring and Observability - Collects and surfaces logs, metrics, and status for every task run to track performance and debug issues.
• Real-Time Monitoring Dashboards - Provides a web dashboard for viewing real-time and historical run, worker, and workflow details.
• Workflow Monitoring Systems - Provides real-time observability through a web UI, OpenTelemetry, Prometheus metrics, and logging.
• Composite - Waits for the first of multiple conditions, like sleep completion or event arrival, before resuming a task.
• External Event Integration - Accepts external webhook events and converts them into workflow triggers for real-time processing.
• Worker Process Executions - Starts a long-running worker process that registers with the engine and executes assigned tasks.
• Worker Process Executions - Starts worker processes that register tasks and workflows for execution.
• Agent Observability Tools - Displays each agent tool call as a task run with status, timing, and input/output details in the dashboard.
• Agent Tool Definitions - Generates provider-specific tool definitions from Hatchet workflow or task definitions for use with an agent SDK.
• Workflow Tool Exposures - Makes Hatchet tasks available as tools for AI agents through MCP or SDKs, adding durable execution and retries to each tool call.
• AI Agent Tool Integrations - Exposes durable tasks as callable tools for AI agents through MCP or SDKs with retries and observability.
• Agent Tool Call Persistences - Persists execution state across worker restarts and failures for every run submitted by an agent tool.
• Execution Checkpointing - Automatically saves agent state on errors and resumes execution from the last checkpoint for fault tolerance.
• In-Process MCP Server Implementations - Groups Hatchet-backed tools into an MCP server that runs inside the agent process for direct tool discovery and invocation.
• Workflow-as-a-Tool Exposure - Registers a Hatchet workflow with a description and input schema so an AI agent can invoke it as a tool.
• Standalone Task Exposures - Registers a single Hatchet task with a description and input schema so an AI agent can invoke it as a tool.
• Workflow Definitions - Lists, updates, and removes workflow definitions through a dedicated workflows client.
• Tenant Workflow Listings - Lists all workflow declarations within a tenant with pagination and name filtering.
• Exactly-Once Processing Semantics - Guarantees each task step runs exactly once via checkpointing and replay from the last checkpoint on failure.
• Workflow Deletions - Removes workflow declarations from the system by name, ID, or object reference.
• Conditional Task Skipping - Conditionally skips or cancels tasks based on parent task output at runtime.
• CLI Workflow Integrations - Develops, triggers, and debugs workflows from the terminal.
• Cron Trigger Management - Creates, updates, and deletes cron-triggered workflows through a dedicated client interface.
• Programmatic Cron Triggers - Creates cron triggers via the API with a name and expression, enabling dynamic scheduling based on business logic.
• Worker-Task Routing - Routes tasks to specific workers using labels, affinity rules, or weighted scheduling.
• Execution Policies - Applies concurrency limits, rate limits, and priorities to control task assignment and execution.
• Task Execution Prioritization - Assigns priority levels so critical tasks run before less latency-sensitive ones like backfills.
• Composite Wait Conditions - Combines parent dependencies, sleep timers, and external signals with OR/AND logic to control task start conditions.
• Failed Task Inspections - Provides a dashboard for inspecting failed task runs and manually retrying them.
• One-Time Task Executions - Triggers a task to run at a specific future time by enqueuing it when the clock reaches that moment.
• Production Control Plane Deployments - Deploys a production-grade multi-container stack with PostgreSQL and RabbitMQ.
• Engine Instance Replication - Runs several Hatchet engine replicas behind a load balancer to keep the system available during infrastructure failures.
• Event Filtering Rules - Matches incoming event payloads against CEL expressions to resume tasks only when expected data arrives.
• Execution Time Limits - Sets maximum execution duration for tasks, treating timeout as a failure for retry.
• Execution Timeouts - Controls task execution and queue wait timeouts with configurable limits.
• Delayed Executions - Enqueues a task to run at a specific future time, persisting the delay in the engine rather than worker memory.
• Kubernetes Deployments - Deploys the full Hatchet stack on Kubernetes using a Helm chart with automatic key generation.
• Multi-Instance Deployments - Runs multiple engine instances behind a load balancer for high availability during failures.
• Managed Orchestration Services - Provides a managed control plane that handles upgrades, scaling, and backups automatically.
• Rate Limiters - Configures and inspects rate limits that cap how many tasks a resource can execute in a time window.
• Task Throughput Rate Limiters - Restricts task processing rate per worker or dynamic key within a time window.
• Resource Caps - Enforces per-tenant caps on workers, events, and task runs with alarm thresholds.
• Bulk Task Executions - Executes a single task multiple times with different inputs in one call.
• Service Splitting Deployments - Splits the engine, API, and frontend into separate deployments for independent scaling.
• Evicted Task Resumptions - Restore an evicted task on a worker when its wait condition is satisfied, replaying its event log from the last checkpoint.
• Workflow Run Management - Lists, cancels, and replays individual task and workflow runs through a dedicated runs client.
• Instance Cancellation - Cancels workflow runs programmatically via the runs client or dashboard with cooperative cancellation flags.
• Agent-Triggered Run Inspections - Inspects tool-triggered runs with status, timing, logs, and OpenTelemetry traces in the Hatchet dashboard.
• Workflow Lifecycle Hooks - Defines hooks that run after a workflow succeeds or fails.
• Scheduled Task Cancellation - Removes a future task from the queue so it never runs.
• Task Priority Management - Assigns priority levels to tasks so critical work runs before less urgent ones.
• Bulk Cancellations - Stops multiple task runs at once by providing a list of run IDs or a set of filter criteria.
• Worker gRPC Communication Encryptions - Applies TLS or mTLS to all gRPC connections between the worker and the engine to secure data in transit.
• Webhook Signature Verifiers - Validates incoming webhook requests using a shared secret to ensure they originate from the expected source.
• Encrypted Persistence - Protects stored secrets and tokens using local keys, a file-based keyset, or Google Cloud KMS before persisting them to the database.
• Worker Payload End-to-End Encryptions - Encrypts task inputs before they leave the worker and decrypts outputs upon return, keeping plaintext data within the worker.
• gRPC and HTTP TLS Securings - Encrypts all server-to-server and client-to-server communication using TLS with configurable certificates, minimum version, and internal client settings.
• Human-in-the-Loop Workflows - Pauses workflows for human input or external events and resumes from checkpoints without custom recovery logic.
• Agent Tool Rate Limits - Limits the rate and concurrency of agent-triggered runs to protect downstream systems from bursts.
• Programmatic Workflow Authoring - Lists, inspects, and controls workflow definitions through a dedicated workflows client.
• Entity-Scoped Event Conditions - Restricts workflow event waits to events with a matching entity identifier.
• Agent-Triggered Task Retries - Automatically retries failed agent tool calls according to the workflow or task retry configuration.
• Manual Task Retries - Allows manual retry of individual failed tasks from the dashboard with original inputs.
• Worker Slot Allocations - Limits how many tasks a worker processes simultaneously using configurable slots, queuing excess tasks until a slot opens.
• Workflow Declaration Retrievals - Fetches a single workflow declaration by name, ID, or object reference.
• Failure-Triggered Tasks - Triggers designated tasks only when other tasks in the workflow fail.
• Success-Triggered Tasks - Triggers designated tasks only when all tasks in the workflow succeed.
• Workflow Triggers - Runs a workflow defined in the project's configuration file from the command line.
• Debugging Observers - Persists every task and agent invocation in a durable event log for real-time monitoring, debugging, and replay.
• OpenTelemetry Exporters - Ships built-in OpenTelemetry exporters for distributed traces and Prometheus metrics.
• Workflow Management Dashboards - Provides a web interface for monitoring workflow runs, logs, metrics, and traces with real-time status and debugging capabilities.
• Prometheus-Based Metric Exporters - Exposes Prometheus metrics at a configurable path with optional tenant scoping.
• Remote Task Cancellation - Stops running tasks programmatically via client or dashboard with cooperative cancellation flags.
• Programmatic Worker Management - Lists, inspects, and controls worker instances through a dedicated workers client API.
• Metadata Filtering - Filters events and task runs in the dashboard by attached metadata key-value pairs.
• Retry and Backoff Logic - Implements exponential backoff for retrying failed tasks, increasing delay between retries to allow service recovery.
• Data Structures - Distributed, fault-tolerant task queue.

Star history