kubeflow/pipelines

Features

• Machine Learning Orchestration - Provides full orchestration of the end-to-end machine learning lifecycle by coordinating sequential and parallel tasks on a cluster.
• ML Workflow Orchestrations - Coordinates the end-to-end process of designing and deploying machine learning pipelines to automate the model lifecycle.
• Kubernetes-Native Workflows - Orchestrates the ML workflow lifecycle natively on Kubernetes by coordinating pod specifications and task execution.
• Distributed ML Task Automation - Automates the end-to-end execution of machine learning tasks on a cluster for portable and scalable deployments.
• Distributed ML Trainers - Implements scalable systems for distributing the construction of machine learning models across compute clusters.
• ML Pipeline Orchestration - Provides tools for designing and managing end-to-end machine learning workflows including task dependencies and scheduling.
• DSL-to-Specification Compilation - Transforms a domain-specific language into structured pipeline specifications to execute complex workloads across a distributed cluster.
• ML Pipeline Automation - Enables the design and deployment of scalable machine learning pipelines using a domain-specific language.
• Pipeline Artifact Passing - Implements strongly typed artifacts to transfer immutable data and metadata between sequential pipeline stages.
• Pipeline Artifact Managers - Manages the flow of strongly typed artifacts and custom metadata between different pipeline components.
• ML Workflow Engines - Ships an engine that manages the packaging and execution of machine learning components in containerized pipelines.
• Workflow Lifecycle Management - Manages the complete workflow lifecycle, including parameter resolution, pod specification patching, and artifact transfers.
• Pipeline Compilers - Compiles Python-defined machine learning workflows into structured specifications for portable and scalable deployment.
• Cluster Coordination - Coordinates the synchronization of state and workload distribution across cluster nodes to automate the ML lifecycle.
• Private Multi-Tenant Hosting - Sets up isolated multi-user environments with integrated identity provider authentication on private infrastructure.
• Data Workflow Execution - Provides mechanisms for running data processing pipelines across both local machines and distributed Kubernetes clusters.
• DSL Compilers - Compiles a Python-based domain-specific language into structured YAML specifications for distributed cluster execution.
• Namespace-Based Isolation - Ensures multi-user isolation and security by restricting resource access to specific Kubernetes namespaces.
• Container-Based Isolation - Executes individual pipeline steps within isolated containers to ensure consistent runtimes and resource enforcement.
• Experiment Tracking - Provides tools for logging, versioning, and visualizing machine learning model training and evaluation workflows.
• ML Asset Versioning - Tracks and manages versions of workflow specifications to ensure reproducible machine learning runs.
• Execution Result Caches - Reuses previous task outputs based on input values to avoid redundant computations and accelerate execution.
• Persistent Storage Volumes - Connects persistent volume claims to pipeline tasks for shared data access and large-scale storage.
• Parallel Task Execution - Executes multiple pipeline steps concurrently while applying resource limits to manage system consumption.
• Pipeline Version Storages - Tracks multiple versions of pipeline definitions and enables setting a default version for execution.
• Pipeline Specification Generation - Transforms code into workflow specifications in YAML or compressed formats for deployment.
• Content-Addressable Caches - Avoids redundant computations by reusing previous task outputs based on a hash of input values.
• Multi-Tenant ML Platforms - Provides a scalable environment for managing ML experiments with isolated namespaces and integrated identity provider authentication.
• Reusable Pipeline Components - Allows storing and retrieving modular workflow building blocks to assemble new solutions without rebuilding steps.
• Pipeline Run Schedulers - Enables the triggering of pipeline executions on a fixed timetable using the latest available pipeline version.
• Hardware Resource Allocation - Allows specifying CPU, memory, and GPU requirements for tasks to ensure appropriate hardware allocation on the cluster.
• Workflow Conditional Logic - Evaluates runtime values to branch pipelines and execute different workflow paths through conditional components.
• Workflow Orchestration - Standardized pipeline orchestration for the Kubeflow ecosystem.
• GitOps And Continuous Delivery - Deploy portable and scalable machine learning workflows on Kubernetes.
• General Purpose Orchestration - End-to-end ML workflow orchestration for Kubernetes.

Star history