PrefectHQ/prefect

Features

• Data Pipeline Orchestration - Orchestrates complex data pipelines by defining, scheduling, and monitoring task dependencies as code.
• Workflow Orchestration - Provides a comprehensive platform for defining, scheduling, and monitoring complex data pipelines using Python-based code.
• Data Pipeline Orchestrators - Manages task dependencies, retries, and state transitions for resilient and observable data processing workflows.
• Container-Native Infrastructure - Wraps individual tasks in isolated container environments to ensure consistent dependencies and resource allocation.
• Event-Driven Workflow Triggers - Initiates or resumes pipeline execution automatically in response to system state changes or external signals.
• Workflow Event Triggers - Triggers automated workflows in response to system events, schedules, or external signals.
• Workflow Monitoring Systems - Provides a centralized dashboard to track task status, inspect logs, and manage workflow triggers.
• Data Lineage - Maintains and visualizes upstream dependency chains by registering data assets within the system.
• Distributed Task Schedulers - Orchestrates and distributes complex data processing workflows across computing clusters for parallel execution.
• Distributed Task Queues - Distributes parallel workloads across clusters or remote nodes to prevent serial execution bottlenecks and enable horizontal scaling.
• Distributed Task Schedulers - Decouples task definition from execution, allowing remote workers to poll for and process work units across diverse infrastructure.
• Decoupled Architectures - Separates task scheduling from execution by allowing remote workers to poll a central API for pending work units.
• State Machine Orchestrators - Tracks workflow execution progress through discrete state transitions and persistent event logs to ensure reliable task processing.
• Workflow Management Dashboards - Provides a centralized dashboard for monitoring and managing workflow execution.
• Workflow Execution Engines - Runs individual workflow tasks inside isolated containers by polling work pools for pending requests.
• Node Result Caches - Avoids re-running data transformation steps by caching node results when logic and data remain unchanged.
• Database Connection Managers - Store and retrieve database connection credentials as reusable configuration objects to authenticate tasks without local profile files.
• Task Result Storage - Persists task and flow return values to external storage for caching, retries, and cross-process retrieval.
• Workflow Orchestrators - Executes DAGs by parsing manifests and running nodes as individual tasks to enable granular retries and observability.
• Workflow Result Retrieval - Provides mechanisms for fetching outputs from completed asynchronous workflow executions using unique identifiers.
• Background Task Schedulers - Provides background task scheduling to offload long-running operations from web request cycles to dedicated workers.
• Custom Parallel Task Execution - Distributes workflow tasks across clusters to enable concurrent execution and horizontal scaling.
• Parallel Task Orchestrators - Distributes and manages parallel execution of tasks across remote infrastructure to accelerate large-scale data operations.
• Remote Repository Code Fetchers - Provides utilities for retrieving source code from remote version control systems for automated execution.
• Reusable Configuration Blocks - Stores and injects credentials, connection parameters, and infrastructure settings into workflows through reusable, decoupled configuration objects.
• Containerized Task Execution - Enables running workflow tasks inside isolated container environments.
• Containerized Task Execution - Packages workflow code into container images to ensure consistent and reproducible execution environments.
• Workflow Execution Scaling - Scales workflow execution across distributed nodes and clusters.
• Deployment Workflows - Packages and registers workflows for execution in remote environments to enable scalable pipeline operations.
• Distributed Task Orchestrators - Distributes and runs computational workloads across diverse infrastructure, including cloud environments and container clusters.
• Kubernetes Job Execution - Provides native integration for running workflow tasks as isolated Kubernetes jobs.
• Kubernetes Job Orchestration - Orchestrates workflow tasks as isolated Kubernetes jobs.
• Kubernetes Work Pool Execution - Integrates workflow execution with Kubernetes work pools for scalable infrastructure.
• Heartbeat Monitors - Monitors heartbeat signals to detect and mark unresponsive workflows as failed, preventing resource leakage.
• API Clients - Executes operations against orchestration services using synchronous or asynchronous clients to manage workflows.
• AWS Authentication Strategies - Manages secure access to cloud services by storing and retrieving credentials for use across infrastructure and data operations.
• Cloud Secret Managers - Creates, reads, updates, and deletes secrets within cloud provider secret management services using authenticated credentials.
• AWS Role Assumption - Configures temporary security credentials by requesting access to specific cloud roles with custom session durations and policies.
• Audit Logging Systems - Provides historical audit trails and logs of system activity to support monitoring and debugging of pipeline execution.
• Job Monitoring Tools - Triggers remote data transformation jobs and monitors their status with automatic retries.
• Workflow Execution APIs - Provides APIs to query, filter, and update the execution status of workflow runs.
• Cloud Workflow Runners - Deploys and runs workflow tasks within managed cloud container environments with execution monitoring.
• Workflow Orchestration - Workflow automation platform for data and ML tasks.
• Data Pipelines - Adds reliability features like retries and caching to pipelines.
• Data Engineering - Workflow orchestration for resilient data pipelines.
• DevOps and Infrastructure - Workflow orchestration for data pipelines.
• Job Schedulers - Workflow orchestration for data pipelines.
• General Purpose Orchestration - Modern workflow management system for data-intensive infrastructure.
• Cloud Authentication - Stores and manages access credentials for external cloud providers to securely authorize automated workflows to interact with remote infrastructure.
• Database Connection Managers - Stores and retrieves secure connection details for external database services using reusable blocks managed through a centralized interface.
• Data Pipeline Configurations - Configuration schemas for defining data loading and transformation workflows.
• Result Caching - Saves task outputs to external storage to enable result reuse and optimization of redundant computations.
• Transformation Execution - Executes data transformation commands while automatically capturing logs and tracking assets for visibility.
• Cloud Data Warehouse Connectivity - Provides secure connectivity for executing SQL queries and managing datasets in cloud data warehouses.
• Database Connections - Lifecycle management for opening and initializing database connections.
• Artifact Management - Tools for transferring, logging, and organizing files and directories associated with workflow runs.
• Transformation Asset Trackers - Automatically derives lineage and metadata for transformation models and sources.
• Repository Integrators - Fetches and manages external source code repositories locally for use in automated workflows.
• Task Scheduling - Automates the execution of workflows on specific timetables or cron intervals for consistent data processing.
• Task Lifecycle Management - Manages the lifecycle of containerized task execution environments.
• Azure Container Task Execution - Executes tasks within isolated Azure container instances.
• Azure Container Execution - Supports running data pipelines on Azure container instances.
• Elastic Container Service Execution - Executes data pipeline tasks on managed elastic cloud container services.
• Batch Workload Execution - Runs workflow tasks as isolated jobs within managed serverless container environments.
• Containerized Deployments - Deploys containerized infrastructure using pre-configured templates to execute data pipelines in isolated environments.
• Containerized Worker Orchestrators - Provisions and manages scalable execution infrastructure on cloud container services for isolated workflow processing.
• Cloud Execution Environments - Allows scaling workflow execution on managed cloud infrastructure.
• Infrastructure Templates - Provisions cloud resources using pre-configured templates to standardize execution environments.
• Job Execution Engines - Triggers namespaced jobs within clusters and captures logs from individual pods upon completion.
• Job Scheduling - Triggers and manages execution of batch jobs within managed cloud environments.
• Kubernetes API Clients - Libraries and tools for interacting with Kubernetes control planes.
• Kubernetes Cluster Management - Manages network services within Kubernetes clusters to facilitate communication between application components.
• Kubernetes Configurations - Defines connection parameters and execution settings for running workflows within Kubernetes clusters.
• Kubernetes Job Management - Manages Kubernetes Jobs directly from workflow definitions.
• Serverless Function Management - Executes serverless functions with custom payloads and execution contexts for event-driven tasks.
• GitHub Authentication - Stores and manages authentication tokens to securely access GitHub resources and services within automated workflows.
• Observability Pipelines - Tracks execution progress, lineage, and performance metrics of data tasks through persistent logs and centralized monitoring interfaces.
• Task Monitoring - Tracks the status and results of asynchronous background operations using unique identifiers for observability.
• Data Asset Lifecycle Management - Tracks data assets and their metadata throughout the transformation lifecycle.
• Bulk Data Loading - Imports data from local files or cloud storage into database tables with schema validation.
• Cloud File Uploaders - Utilities for transferring local data blobs and files to remote cloud storage.
• Databricks Authentication Managers - Manages secure connections to data environments using personal access tokens or service principal credentials.
• Data Quality Monitors - Records dynamic metrics like row counts and data quality scores during task execution to provide visibility into processed data.
• Memory-Aware - Clears task or flow results from active memory after commitment to storage to prevent system crashes during large-scale data processing.
• Database Command Interfaces - Executes arbitrary Redis commands directly against a database for use in workflow logic.
• Snowflake Connectors - Specialized modules for establishing secure connections to cloud-based data warehouses.
• Rolling Deployment Migrations - Ensures consistent database state during rolling deployments.
• Database Operations - Executes SQL commands and queries against connected databases using synchronous or asynchronous drivers.
• SQL Query Execution - Executes SQL queries against data warehouses and returns results as structured dataframes.
• Database Query Execution - Runs parameterized SQL queries against databases with support for transaction-based execution.
• External Datastore Configurations - Connects the orchestration engine to external databases and Redis instances for state synchronization and task coordination.
• Operation Execution - Executes custom GraphQL queries and mutations against remote services using authenticated credentials.
• Model Lifecycle Hooks - Triggers custom logic at specific points during transformation runs to enable lifecycle-aware pipeline execution.
• Notebook Automation Tools - Automates the execution of remote notebooks by provisioning compute clusters and passing parameters.
• Real-Time Data Streaming - Enables real-time data ingestion into warehouses using streaming APIs.
• Code Repository Integration Tools - Authenticates with version control systems to clone or download workflow code from private repositories.
• Command Execution - Runs transformation commands within workflows with automatic node-to-task mapping and failure handling.
• Containerized Execution - Triggers and manages containerized data processing tasks within isolated serverless environments.
• Storage Bundle Execution - Downloads, extracts, and runs workflow bundles stored in cloud storage using subprocess execution.
• Storage Clients - Configures and initializes storage clients for interacting with cloud-based object storage services.
• Third-Party Service Integrations - Integrates workflow execution with third-party platforms and cloud providers for data movement and task coordination.
• Workflow Parameter Schemas - Enforces data integrity by validating runtime inputs against defined schemas.
• Automated Deployment Pipelines - Integrates workflow deployment into CI processes to automate image building and pipeline registration.
• Cloud Container Execution - Binds flow execution to cloud container instance work pools for remote execution.
• Container Service Execution - Binds workflow execution to cloud container service work pools to manage remote infrastructure.
• Task & Job Management - Provides interfaces to trigger, track, and manage the execution of remote cloud-based jobs.
• Transformation - Groups multiple transformation commands into reusable blocks that manage execution environments and process lifecycles.
• Remote Task Execution Modules - Triggers remote data processing tasks and monitors their execution status until completion.
• Azure Flow Deployments - Executes workflow processes on cloud container instances to dynamically scale infrastructure based on demand.
• Infrastructure Deployment Provisioning - Provisions compute resources for workflow execution.
• S3-Compatible Storage Adapters - Configures persistent storage blocks for reading and writing data to S3-compatible object storage services.
• Deployment Infrastructure - Orchestrates the deployment of server and worker components within a cluster using package management templates.
• Native Execution Offloading - Offloads data transformation execution to managed ephemeral jobs while maintaining local orchestration features.
• Infrastructure Automation Workflows - Provisions and manages the compute resources, secrets, and containerized environments required to execute automated data pipelines.
• Service Connection Registries - Exposes pre-configured connection templates for discovery and reuse across workflow environments.
• Infrastructure Provisioning - Deploys cloud resources using infrastructure-as-code templates to ensure consistent execution environments.
• Service Scaling - Scales API and background services to ensure high availability.
• Remote Deployment Workflows - Fetches workflow code directly from remote repositories during deployment to execute version-controlled tasks.
• Job Execution Strategies - Initiates external data transformation tasks using configurable parameters and metadata.
• Kubernetes Deployment Management - Manages containerized deployments within Kubernetes clusters.
• Pull Request Management - Provides administrative tools to manage pull requests, including reviews and comments, within deployment pipelines.
• Version Control Repository Integrators - Fetches workflow code from secure remote version control systems during deployment using stored credentials.
• S3 Protocol Clients - Implementations for connecting to and managing remote object storage using the S3 protocol.
• Slack Integrations - Connectors for messaging and file exchange on Slack platforms.
• Authentication Clients - Initializes authenticated clients for secure API requests using stored credentials.
• Cloud Service Authentication - Mechanisms for authenticating with cloud provider services.
• Password Access Restrictions - Enforces basic username and password authentication for server access and client communication to secure the API and web interface.
• GitLab Authentication Managers - Stores access tokens in secure blocks to authorize the retrieval of code from private version control systems.
• External Service Authorizers - Stores and manages API keys to securely authorize connections to third-party data transformation platforms within automated workflows.
• Asset Dependencies - Allows users to define formal relationships between data assets to ensure correct execution order.
• Execution Pausing - Allows pausing workflow execution to await manual approval or external data input before resuming.
• ECS Work Pools - Organizes and manages execution resources for workflow deployments within elastic cloud environments.
• Repository Metadata - Retrieves metadata for repositories using authenticated GraphQL queries.
• Managed AI Job Execution - Runs data pipeline tasks as managed custom jobs within cloud provider environments.
• Workflow Task Definitions - Imports and runs standardized tasks from external service packages to interact with third-party infrastructure.
• IP Access Restrictions - Limits inbound traffic to APIs and interfaces to specific IP addresses or CIDR ranges to enforce strict network perimeter security.
• Execution Run APIs - Retrieves metadata and execution logs for specific data transformation job runs from remote cloud providers.
• Kubernetes Monitors - Analyzes pod status data to identify failure conditions and provide actionable resolution steps for Kubernetes-based tasks.
• Remote Workflow Operations - Downloads and executes serialized workflow bundles from cloud storage to trigger automated data processing.
• Repository Resource Managers - Performs programmatic actions on repositories including issue and pull request management.
• Cloud API Request Execution - Sends authenticated HTTP requests to arbitrary API endpoints to retrieve data or trigger remote operations.
• ECS Worker Services - Provisions containerized worker instances within an ECS cluster for secure workflow execution.
• Issue Tracking - Provides programmatic management of repository issues to automate project tracking and communication.
• Cloud Storage APIs - Provides programmatic access to cloud-based object storage for file organization and data persistence.
• Cloud Storage Exporters - Connectors for pushing processed data to remote object storage buckets.
• Cloud Storage Services - Provides interfaces for managing remote object storage containers and data persistence.
• Cloud Storage Sync Tools - Synchronizes local project files with cloud storage buckets to facilitate deployment.
• Connection Managers - Provides persistent connection management for databases with automatic resource cleanup.
• Transformation Plan Previewers - Computes and validates execution graphs and selector resolution without triggering actual model builds.
• Redis Clients - Provides clients for writing data to Redis with support for expiration and conditional logic.
• Object-to-Redis Mappers - Supports automatic reconstruction of serialized objects when reading data from Redis databases.
• Configuration Importing - Loads database connection configurations directly from project files to maintain environment consistency.
• Configuration Profiles - Tools for applying predefined sets of system settings and optimizations.
• Repository Cloning Tools - Tools for initializing local copies of remote repositories with automated configuration.
• Shell Command Runners - Executes arbitrary shell commands within workflows with custom environment and output handling.
• Task Configuration Overrides - Enables overriding default task settings to adapt pipeline operations to specific requirements.
• Background Service Isolation - Isolates orchestration tasks to dedicated processes for resource efficiency.
• Cloud Service Connectivity Modules - Modules for managing authentication and network connectivity to cloud-hosted services.
• Cluster Job Schedulers - Connects to existing distributed computing clusters to execute workflow tasks.
• Environment Variable Configurations - Maps configurable variables to resource settings like environment variables and secrets for containerized execution.
• Docker Image Building - Creates container images from local configurations to prepare isolated environments for workflow execution.
• Container Image Registry Uploads - Uploads container images to remote registries for deployment.
• Deployment Automation - Orchestrates resource provisioning and flow deployments using programmable infrastructure workflows.
• Orchestration Service Connections - Facilitates communication with centralized orchestration servers from lightweight or ephemeral execution environments.
• Job Environment Management - Defines custom Kubernetes job templates to control resource allocation and container configurations.
• Distributed Cluster Provisioners - Distributes parallel workloads across compute clusters to prevent serial execution bottlenecks.
• Custom Infrastructure Handlers - Enables the creation of specialized infrastructure handlers to manage execution on platforms not natively supported.
• Worker Scaling - Adjusts the number of active worker nodes dynamically based on workload demand to optimize resource usage.
• Workflow Termination Controls - Stops in-progress tasks by signaling infrastructure to shut down and release resources.
• Custom Request Headers - Capabilities for injecting user-defined metadata into HTTP request headers.
• Email Notification Services - Dispatches automated email notifications via SMTP with support for HTML content and file attachments.
• SMTP Credential Managers - Stores and manages SMTP server credentials to enable secure communication and automated email notifications.
• Proxy Traffic Management - Routes and secures external traffic for self-hosted deployments.
• Cross-Site Request Forgery Protections - Enables token-based validation for state-changing requests to protect the server from unauthorized cross-site actions.
• Slack Credential Managers - Stores and retrieves authentication tokens securely to authorize automated interactions with Slack services during pipeline execution.
• Bitbucket Access Managers - Stores tokens, usernames, and passwords to enable secure access to private repositories.
• Registry Authentication Providers - Stores and manages registry credentials to securely authenticate container runtime clients when pulling or pushing images.
• Concurrent Task Limiters - Limits concurrent task execution using shared state to manage resource consumption across distributed workers.
• Data Transformation Pipelines - Provides administrative interfaces to trigger and monitor cloud-based data transformation workflows.
• Lifecycle Hooks - Allows registration of custom logic to execute before or after specific workflow events.
• Webhook Event Notifications - Sends formatted messages and interactive blocks to Slack channels to notify teams about workflow events.
• Atomic Write Coordinators - Groups multiple task executions into a single transaction to ensure that cached results are only saved if every step completes successfully.
• Workflow Input Schemas - Generates interactive, validated user interfaces for workflow parameters using type annotations.
• Database Monitoring - Tracks active and idle database connection counts to prevent service exhaustion and ensure reliable availability.
• Execution Logs - Captures and includes log output from third-party libraries within workflow execution logs for improved debugging.
• GraphQL Data Fetching - Retrieves profile information and activity metadata using authenticated GraphQL queries.

Star history