flyteorg/flyte

Features

• Workflow Orchestration - Coordinates machine learning pipelines using DAG-based orchestration across distributed compute resources for resilience and scalability.
• Data Pipeline Automation - Orchestrates complex task dependencies and schedules recurring jobs for large-scale data processing.
• AI Workflow Orchestrators - Provides a resilient engine for defining and executing AI pipelines with parallel tasks and branching logic.
• Data Lineage - Tracks versions of input and output data to ensure the reproducibility of machine learning workflows.
• Resource Allocation - Dynamically assigns CPU and GPU resources at the task level, including support for spot and preemptible instances.
• Workflow Orchestration Engines - Provides a platform for defining and scheduling complex sequences of data processing tasks and their dependencies.
• Custom Parallel Task Execution - Executes parallel operations in isolated containers with dynamic resource allocation for CPU and GPU workloads.
• Parallel Task Orchestrators - Distributes and manages the parallel execution of tasks across remote infrastructure to reduce processing time.
• Containerized Task Execution - Manages the packaging and execution of individual pipeline tasks within isolated containers.
• Containerized - Packages each task in a separate container to ensure dependency isolation and environment consistency.
• Distributed Task Schedulers - Dynamically assigns CPU and GPU resources across clusters using spot or preemptible instances.
• Kubernetes ML Platforms - Orchestrates machine learning workflows and data pipelines using Kubernetes for distributed compute and scaling.
• Directed Acyclic Graph Engines - Models workflows as directed acyclic graphs to coordinate task execution order and parallelization.
• Task Coordinations - Coordinates the full task lifecycle from resource request and queuing to final completion or failure across distributed nodes.
• ML Pipeline Reproducibility - Tracks data lineage and immutable execution states to ensure reproducible machine learning results.
• Container-Based Isolation - Ensures dependency isolation and environment consistency by packaging individual tasks into separate containers.
• Workflow Execution Managers - Provides a centralized system to track, filter, and abort active workflow runs to prevent redundant processing.
• Model Serving - Provides the infrastructure to deploy trained models as scalable API endpoints for inference.
• Model Serving Transformations - Transforms trained models into active endpoints that serve predictions to users or external applications.
• Task Execution Caches - Implements memoization to store and retrieve previous task outcomes, avoiding redundant expensive computations.
• Cloud-Local Data Interfaces - Facilitates moving datasets between local environments and cloud storage using column-level type checking.
• Real-time Stream Monitors - Provides real-time tailing and filtering of logs generated by active workflow runs for monitoring and debugging.
• Multi-Backend Storage Management - Handles permissions and generates signed URLs for data transfers across multiple cloud storage backends.
• Signed URL Proxies - Provides secure, signed-URL access to blob storage for task input and output operations.
• Job Schedulers - Automates the periodic execution of data pipelines with integrated status alerts.
• Pipeline Execution CLIs - Provides a command line interface to trigger workflow executions and pass input parameters.
• Task Lifecycle Management - Monitors the entire lifecycle of containerized tasks from initial queueing to final completion.
• Granular Task Retries - Enables restarting specific failed operations or rerunning individual steps without resetting the entire workflow.
• Model Endpoint Deployment - Transforms trained machine learning models into scalable API endpoints for production serving.
• Multi-Cloud Deployments - Supports promoting workflows across different development and production domains on multiple cloud platforms.
• Custom Container Integration - Integrates non-native containers by managing metadata downloads and output uploads to remote stores.
• Workflow Event Triggers - Initiates automated workflows based on defined time intervals or specific system events.
• Multi-tenant Security - Provides isolation and access control mechanisms to allow multiple users to share a single platform securely.
• Logical Resource Partitioning - Provides secure logical isolation of configurations and data for multiple users on a shared platform.
• Memoization Caches - Caches the outputs of completed tasks to bypass redundant computations when inputs remain identical.
• Workflow Logic Engines - Supports data-driven branching and pausing for external approvals within automated workflow logic.
• Workflow Task Definitions - Allows the creation and versioning of specifications used to organize unit-level work within durable workflows.
• Execution Progress Tracking - Monitors real-time status and reports the final outcomes of long-running tasks to a centralized coordination service.
• Execution State Streaming - Provides real-time emission of execution status and source locations via streaming RPCs for immediate visibility.
• Workflow Performance Diagnostics - Identifies processing bottlenecks and visualizes training history using task timeline views and performance plots.
• Real-time Status Streaming - Uses streaming RPCs to push real-time execution logs and status changes from workers to the controller.
• Workflow Orchestration - Scalable automation platform for mission-critical ML processes.
• Data Pipelines - Provides a cloud-native platform for ML and data processing.
• MLOps and Workflows - Workflow automation for scalable machine learning and data processing.

Star history