kubeflow/pipelines
Pipelines
Este proyecto es un motor de flujo de trabajo de aprendizaje automático contenerizado y orquestador diseñado para automatizar el ciclo de vida completo de modelos de aprendizaje automático en clusters de Kubernetes. Funciona como un compilador de pipeline de MLOps que transforma un lenguaje específico de dominio en especificaciones estructuradas para un despliegue portátil y escalable.
La plataforma proporciona un entorno multi-inquilino con namespaces aislados y autenticación mediante proveedor de identidad. Se distingue por una combinación de aislamiento de tareas basado en contenedores, gestión de artefactos fuertemente tipados para el paso de datos y caché de resultados direccionable por contenido para evitar cálculos redundantes.
El sistema cubre la orquestación integral de flujos de trabajo, incluyendo ejecución de tareas en paralelo, programación de ejecuciones recurrentes y lógica de ramificación condicional. Además, admite el seguimiento de experimentos, la recolección de métricas de flujo de trabajo y la gestión de componentes de pipeline reutilizables, con la capacidad de configurar solicitudes de recursos de hardware específicos para CPU, memoria y GPU.
El software se distribuye a través de un SDK de Python y puede desplegarse en entornos independientes, locales o multi-inquilino.
Features
- Machine Learning Orchestration - Provides full orchestration of the end-to-end machine learning lifecycle by coordinating sequential and parallel tasks on a cluster.
- ML Workflow Orchestrations - Coordinates the end-to-end process of designing and deploying machine learning pipelines to automate the model lifecycle.
- Kubernetes-Native Workflows - Orchestrates the ML workflow lifecycle natively on Kubernetes by coordinating pod specifications and task execution.
- Distributed ML Task Automation - Automates the end-to-end execution of machine learning tasks on a cluster for portable and scalable deployments.
- Distributed ML Trainers - Implements scalable systems for distributing the construction of machine learning models across compute clusters.
- ML Pipeline Orchestration - Provides tools for designing and managing end-to-end machine learning workflows including task dependencies and scheduling.
- DSL-to-Specification Compilation - Transforms a domain-specific language into structured pipeline specifications to execute complex workloads across a distributed cluster.
- ML Pipeline Automation - Enables the design and deployment of scalable machine learning pipelines using a domain-specific language.
- Pipeline Artifact Passing - Implements strongly typed artifacts to transfer immutable data and metadata between sequential pipeline stages.
- Pipeline Artifact Managers - Manages the flow of strongly typed artifacts and custom metadata between different pipeline components.
- ML Workflow Engines - Ships an engine that manages the packaging and execution of machine learning components in containerized pipelines.
- Workflow Lifecycle Management - Manages the complete workflow lifecycle, including parameter resolution, pod specification patching, and artifact transfers.
- Pipeline Compilers - Compiles Python-defined machine learning workflows into structured specifications for portable and scalable deployment.
- Cluster Coordination - Coordinates the synchronization of state and workload distribution across cluster nodes to automate the ML lifecycle.
- Private Multi-Tenant Hosting - Sets up isolated multi-user environments with integrated identity provider authentication on private infrastructure.
- Data Workflow Execution - Provides mechanisms for running data processing pipelines across both local machines and distributed Kubernetes clusters.
- DSL Compilers - Compiles a Python-based domain-specific language into structured YAML specifications for distributed cluster execution.
- Namespace-Based Isolation - Ensures multi-user isolation and security by restricting resource access to specific Kubernetes namespaces.
- Container-Based Isolation - Executes individual pipeline steps within isolated containers to ensure consistent runtimes and resource enforcement.
- Experiment Tracking - Provides tools for logging, versioning, and visualizing machine learning model training and evaluation workflows.
- ML Asset Versioning - Tracks and manages versions of workflow specifications to ensure reproducible machine learning runs.
- Execution Result Caches - Reuses previous task outputs based on input values to avoid redundant computations and accelerate execution.
- Persistent Storage Volumes - Connects persistent volume claims to pipeline tasks for shared data access and large-scale storage.
- Parallel Task Execution - Executes multiple pipeline steps concurrently while applying resource limits to manage system consumption.
- Pipeline Version Storages - Tracks multiple versions of pipeline definitions and enables setting a default version for execution.
- Pipeline Specification Generation - Transforms code into workflow specifications in YAML or compressed formats for deployment.
- Content-Addressable Caches - Avoids redundant computations by reusing previous task outputs based on a hash of input values.
- Multi-Tenant ML Platforms - Provides a scalable environment for managing ML experiments with isolated namespaces and integrated identity provider authentication.
- Reusable Pipeline Components - Allows storing and retrieving modular workflow building blocks to assemble new solutions without rebuilding steps.
- Pipeline Run Schedulers - Enables the triggering of pipeline executions on a fixed timetable using the latest available pipeline version.
- Hardware Resource Allocation - Allows specifying CPU, memory, and GPU requirements for tasks to ensure appropriate hardware allocation on the cluster.
- Workflow Conditional Logic - Evaluates runtime values to branch pipelines and execute different workflow paths through conditional components.
- Workflow Orchestration - Standardized pipeline orchestration for the Kubeflow ecosystem.
- General Purpose Orchestration - End-to-end ML workflow orchestration for Kubernetes.
