Kestra

Kestra is a declarative workflow orchestrator designed to manage complex task dependencies and automated processes through versioned configuration files. It functions as a distributed platform that decouples task scheduling from execution by offloading computational workloads to a fleet of worker nodes. The system uses a reactive, event-driven engine to initiate workflows automatically in response to external signals, webhooks, schedules, or file system changes.

The platform distinguishes itself through a modular plugin architecture that allows for the integration of custom tasks and external services. It provides an AI-native development environment that incorporates language models to generate, refine, and execute automation logic using natural language prompts. To support diverse operational needs, Kestra implements a multi-tenant execution model that isolates resources, data, and access controls for different teams within a single shared instance.

The system covers a broad range of operational capabilities, including robust state management, granular role-based access control, and comprehensive system auditing. It offers extensive tools for workflow logic, such as conditional branching, parallel task execution, and iterative processing, alongside built-in resilience features like automated retries and failure policies. Users can manage these configurations through a centralized interface that supports visual editing and real-time monitoring of execution status.

Features

Data Pipeline Orchestrators - Automates complex sequences of data processing tasks, including scheduling and dependency management.
Workflow Orchestrators - Manages complex task dependencies and execution logic through versioned configuration files.
Workflow Orchestration - Kestra groups tasks to run one after another in a defined sequence, allowing downstream tasks to access outputs from preceding steps.
AI-Native Development Environments - Enables users to generate, refine, and execute workflow code using natural language prompts.
Workflow Automation - Triggers business processes automatically in response to external events and scheduled intervals.
Event-Driven Automation Engines - Triggers and executes automated processes based on schedules, webhooks, or external system events.
Workflow Event Triggers - Sets up automated execution using schedules, webhooks, or event-based triggers to initiate processes.
Declarative Configuration - Defines complex workflow logic and task dependencies using versioned, human-readable configuration files.
Workflow Definitions - Defines orchestration units using configuration files to manage tasks, inputs, and execution logic.
AI-Native Development - Uses natural language prompts to generate, refine, and execute complex automation logic.
Distributed Task Queues - Offloads computational workloads to a fleet of worker nodes via a centralized message bus.
Event-Driven Triggers - Initiates workflow execution automatically by monitoring external signals, webhooks, and schedules.
Orchestration Logic - Manages workflow execution using control-logic tasks that handle parallel execution, conditional branching, and iterative processing.
Workflow Triggers - Initiates workflow runs automatically based on external events, scheduled times, webhooks, or file system changes.
Data Processing Tasks - Offloads compute-intensive data tasks like file operations and API requests to dedicated worker components.
State Management - Maintains workflow execution status and metadata in a centralized database for auditability and reliability.
Workflow Parameter Schemas - Defines strongly-typed inputs with validation and default values to ensure parameters are provided before execution.
Infrastructure Orchestration - Coordinates microservices and infrastructure operations through versioned, declarative configuration files.
Execution Isolation - Provides secure task execution isolation to prevent cross-tenant interference.
Multi-Tenancy Management - Isolates resources, data, and access controls for different teams within a single shared instance.
Role-Based Access Control - Assigns specific permissions to users and service accounts to control access to resources.
Conditional Branching - Routes workflow logic to specific tasks based on the evaluation of contextual variables.
Task Execution Engines - Runs individual actions within a flow, choosing between computational tasks for data work or flowable tasks for controlling execution logic.
Workflow Replay Systems - Manages failed workflows by restarting specific tasks or replaying from a point of failure.
Audit Logging - Tracks all actions performed by users and service accounts to ensure accountability and compliance.
Workflow Orchestration - Declarative, event-driven platform for managing complex workflows.
Data Engineering - Event-driven orchestrator for data workflow management.
DevOps and Infrastructure - Workflow orchestration platform.
Workflow Orchestration - Workflow orchestration.
General Purpose Orchestration - Data orchestration platform featuring a declarative YAML syntax.
Workflow Frameworks - Data orchestration platform with declarative syntax.
Task Result Storage - Saves task results in internal storage to generate a file URI that other tasks can reference.
Plugin Architectures - Enables integration of custom tasks and external services through a modular plugin interface.
Task Scheduling Policies - Assigns specific tasks to designated worker groups to isolate resource-intensive workloads.
Multi-Tenancy Governance - Manages isolated workflows and resources for different teams while maintaining strict security boundaries.
Secret Management - Integrates external services to store and retrieve sensitive data securely during task execution.
Parallel Execution Engines - Runs multiple tasks simultaneously to improve processing efficiency in distributed environments.
Task Retry Policies - Defines automatic retry policies for failed tasks, including maximum attempts and interval strategies.
Collection Iterators - Processes lists of values by executing task groups for each item with support for concurrency limits.
Inter-flow Data Sharing - Shares typed data between parent and sub-flows, allowing downstream tasks to access results from child executions.
Execution Rate Limiters - Restricts the number of simultaneous executions for a specific flow to protect downstream systems.
Environment Isolation - Separates resources, data, and access controls for different teams using isolated administrative environments.
Execution Sandboxes - Ensures security by running individual tasks within isolated, containerized environments.
Single Sign-On - Authenticates users through external identity providers using the OpenID Connect protocol.
Error Handling Workflows - Defines a list of tasks to execute sequentially when an error occurs to perform automated alerts or cleanup.
Execution Failure Policies - Defines task execution policies to allow downstream tasks to continue running despite failures.
Subflow Orchestrators - Executes child flows from parent workflows with support for completion waiting and state propagation.
Task Property Templating - Sets task properties using static values or dynamic expressions while respecting data type constraints.
Workflow Input Schemas - Provides strongly typed parameters at execution time to customize flow behavior.
Dynamic Data Accessors - References dynamic iteration values and task outputs within loops to process list elements or results.
Execution Context Management - Updates execution-scoped variables during a workflow run to inject new data into the context.
Template Rendering Engines - Generates dynamic values and controls template flow using expression and logic delimiters.
Concurrency Policies - Defines how the system handles new executions when limits are reached by queuing, canceling, or failing requests.
Incident Containment - Stops problematic executions across specific tenants to contain incidents without pausing the entire platform.
Trigger Condition Filters - Restricts workflow execution based on criteria like status, labels, or custom expressions.
Data Encryption - Kestra protects sensitive script task outputs by encrypting them, ensuring they remain hidden in the interface while remaining accessible to subsequent tasks.
Service Account Management - Provides programmatic API access for external applications using assigned roles and tokens.
Configuration Variables - Enables dynamic injection of static and computed values into workflow configurations using templating syntax.
Event-Driven Triggers - Initiates separate, decoupled flows from current executions to enable modular process design.
Global Retry Strategies - Defines system-wide retry policies to automatically apply consistent error recovery behavior across all tasks.
Namespace Management - Organizes secrets, variables, and plugin defaults within logical namespaces for fine-grained access control.
Subflow Data Interfaces - Passes input arguments to child flows and retrieves output values for use in downstream tasks.
Task Identity Management - Assigns core properties to tasks to uniquely identify each step and specify its implementation class.

PrefectHQ/prefect

21,640View on GitHub

Prefect is a workflow orchestration platform designed to define, schedule, and monitor complex data pipelines as Python code. It functions as a container-native engine that wraps individual tasks in isolated environments, ensuring consistent dependencies and resource allocation across diverse infrastructure. By utilizing a state-machine-based orchestration model, the system tracks execution progress through discrete transitions and persistent event logs to maintain reliable and observable task processing. The platform distinguishes itself through a decoupled worker-API architecture, which sep

apache/airflow

45,902View on GitHub

Airflow is a platform for programmatically authoring, scheduling, and monitoring complex data pipelines. It functions as a workflow automation engine that manages the lifecycle of recurring business processes by executing code-defined task dependencies. By representing workflows as directed acyclic graphs, the system ensures that task execution order and data flow are explicitly defined and reliably maintained across distributed computing environments. The platform distinguishes itself through a highly modular, provider-based architecture that decouples core orchestration logic from external

spotify/luigi

18,676View on GitHub

Luigi is a Python framework designed for building and managing complex batch data pipelines. It functions as a workflow orchestration engine that organizes tasks into directed acyclic graphs, ensuring that jobs execute in the correct logical order based on their dependencies. By utilizing a centralized scheduler, the system coordinates task execution across distributed environments, tracks global workflow state, and prevents redundant processing by verifying the existence of output targets before triggering any work. The project distinguishes itself through a robust state-tracking mechanism t

j3ssie/Osmedeus

6,425View on GitHub

Osmedeus is a security workflow orchestration engine that coordinates AI agents, shell commands, and scanning tools through declarative YAML pipelines. It functions as a distributed security scanner, a declarative workflow automator, and an AI agent framework for security, enabling automated multi-step security analysis with conditional branching, parallel execution, and distributed workers. The engine distinguishes itself through a hybrid runner model that executes workflow steps on the local host, inside Docker containers, or over SSH to remote machines, selected per step or module. It supp

kestra-iokestra

Features