kestra-io/kestra

Features

• Data Pipeline Orchestrators - Automates complex sequences of data processing tasks, including scheduling and dependency management.
• Workflow Orchestrators - Manages complex task dependencies and execution logic through versioned configuration files.
• Workflow Orchestration - Kestra groups tasks to run one after another in a defined sequence, allowing downstream tasks to access outputs from preceding steps.
• AI-Native Development Environments - Enables users to generate, refine, and execute workflow code using natural language prompts.
• Workflow Automation - Triggers business processes automatically in response to external events and scheduled intervals.
• Event-Driven Automation Engines - Triggers and executes automated processes based on schedules, webhooks, or external system events.
• Workflow Event Triggers - Sets up automated execution using schedules, webhooks, or event-based triggers to initiate processes.
• Declarative Configuration - Defines complex workflow logic and task dependencies using versioned, human-readable configuration files.
• Workflow Definitions - Defines orchestration units using configuration files to manage tasks, inputs, and execution logic.
• AI-Native Development - Uses natural language prompts to generate, refine, and execute complex automation logic.
• Distributed Task Queues - Offloads computational workloads to a fleet of worker nodes via a centralized message bus.
• Event-Driven Triggers - Initiates workflow execution automatically by monitoring external signals, webhooks, and schedules.
• Orchestration Logic - Manages workflow execution using control-logic tasks that handle parallel execution, conditional branching, and iterative processing.
• Workflow Triggers - Initiates workflow runs automatically based on external events, scheduled times, webhooks, or file system changes.
• Data Processing Tasks - Offloads compute-intensive data tasks like file operations and API requests to dedicated worker components.
• State Management - Maintains workflow execution status and metadata in a centralized database for auditability and reliability.
• Workflow Parameter Schemas - Defines strongly-typed inputs with validation and default values to ensure parameters are provided before execution.
• Infrastructure Orchestration - Coordinates microservices and infrastructure operations through versioned, declarative configuration files.
• Execution Isolation - Provides secure task execution isolation to prevent cross-tenant interference.
• Multi-Tenancy Management - Isolates resources, data, and access controls for different teams within a single shared instance.
• Role-Based Access Control - Assigns specific permissions to users and service accounts to control access to resources.
• Conditional Branching - Routes workflow logic to specific tasks based on the evaluation of contextual variables.
• Task Execution Engines - Runs individual actions within a flow, choosing between computational tasks for data work or flowable tasks for controlling execution logic.
• Workflow Replay Systems - Manages failed workflows by restarting specific tasks or replaying from a point of failure.
• Audit Logging - Tracks all actions performed by users and service accounts to ensure accountability and compliance.
• Workflow Orchestration - Declarative, event-driven platform for managing complex workflows.
• Data Engineering - Event-driven orchestrator for data workflow management.
• DevOps and Infrastructure - Workflow orchestration platform.
• Workflow Orchestration - Workflow orchestration.
• General Purpose Orchestration - Data orchestration platform featuring a declarative YAML syntax.
• Workflow Frameworks - Data orchestration platform with declarative syntax.
• Task Result Storage - Saves task results in internal storage to generate a file URI that other tasks can reference.
• Plugin Architectures - Enables integration of custom tasks and external services through a modular plugin interface.
• Task Scheduling Policies - Assigns specific tasks to designated worker groups to isolate resource-intensive workloads.
• Multi-Tenancy Governance - Manages isolated workflows and resources for different teams while maintaining strict security boundaries.
• Secret Management - Integrates external services to store and retrieve sensitive data securely during task execution.
• Parallel Execution Engines - Runs multiple tasks simultaneously to improve processing efficiency in distributed environments.
• Task Retry Policies - Defines automatic retry policies for failed tasks, including maximum attempts and interval strategies.
• Collection Iterators - Processes lists of values by executing task groups for each item with support for concurrency limits.
• Inter-flow Data Sharing - Shares typed data between parent and sub-flows, allowing downstream tasks to access results from child executions.
• Execution Rate Limiters - Restricts the number of simultaneous executions for a specific flow to protect downstream systems.
• Environment Isolation - Separates resources, data, and access controls for different teams using isolated administrative environments.
• Execution Sandboxes - Ensures security by running individual tasks within isolated, containerized environments.
• Single Sign-On - Authenticates users through external identity providers using the OpenID Connect protocol.
• Error Handling Workflows - Defines a list of tasks to execute sequentially when an error occurs to perform automated alerts or cleanup.
• Execution Failure Policies - Defines task execution policies to allow downstream tasks to continue running despite failures.
• Subflow Orchestrators - Executes child flows from parent workflows with support for completion waiting and state propagation.
• Task Property Templating - Sets task properties using static values or dynamic expressions while respecting data type constraints.
• Workflow Input Schemas - Provides strongly typed parameters at execution time to customize flow behavior.
• Dynamic Data Accessors - References dynamic iteration values and task outputs within loops to process list elements or results.
• Execution Context Management - Updates execution-scoped variables during a workflow run to inject new data into the context.
• Template Rendering Engines - Generates dynamic values and controls template flow using expression and logic delimiters.
• Concurrency Policies - Defines how the system handles new executions when limits are reached by queuing, canceling, or failing requests.
• Incident Containment - Stops problematic executions across specific tenants to contain incidents without pausing the entire platform.
• Trigger Condition Filters - Restricts workflow execution based on criteria like status, labels, or custom expressions.
• Data Encryption - Kestra protects sensitive script task outputs by encrypting them, ensuring they remain hidden in the interface while remaining accessible to subsequent tasks.
• Service Account Management - Provides programmatic API access for external applications using assigned roles and tokens.
• Configuration Variables - Enables dynamic injection of static and computed values into workflow configurations using templating syntax.
• Event-Driven Triggers - Initiates separate, decoupled flows from current executions to enable modular process design.
• Global Retry Strategies - Defines system-wide retry policies to automatically apply consistent error recovery behavior across all tasks.
• Namespace Management - Organizes secrets, variables, and plugin defaults within logical namespaces for fine-grained access control.
• Subflow Data Interfaces - Passes input arguments to child flows and retrieves output values for use in downstream tasks.
• Task Identity Management - Assigns core properties to tasks to uniquely identify each step and specify its implementation class.

Star history