Netflix/metaflow

Features

• Workflow Orchestration - Orchestrates data pipelines using DAGs with support for parallel execution, conditional logic, and remote compute scaling.
• ML Workflow Engines - Provides a specialized engine for developing and executing machine learning data pipelines with integrated support for recursion and branching.
• Machine Learning Pipelines - Orchestrates machine learning pipelines from local prototyping to production with versioned experiments.
• Experiment Tracking - Records versions and tracks parameters to monitor the progress of research and model iterations.
• Experiment Tracking Systems - Records pipeline versions, parameters, and artifacts to monitor research iterations and model performance.
• Python Machine Learning Libraries - Provides a Python-based framework for building and managing reproducible machine learning pipelines.
• Compute Cluster Orchestration - Sets up temporary clusters of communicating instances to support distributed computing tasks.
• Data Checkpointing - Automatically persists intermediate data artifacts to remote storage to allow for failure recovery and inspection.
• Variable Persistence - Saves instance variables to a persistent store to allow data recovery and sharing across environments.
• State Checkpointing - Saves progress periodically during task execution to prevent data loss and allow recovery.
• Workflow Scheduling And Orchestration - Enables automated execution of production pipelines on fixed timetables using external orchestrators.
• Parallel Execution - Supports parallel execution of workflow steps across local CPU cores or remote cloud instances.
• Parallel Task Orchestrators - Runs multiple operations concurrently across data shards using local CPU cores or remote containers.
• Pipeline Structuring - Organizes complex machine learning logic into a structured sequence of top-level flows and individual steps.
• MLOps Pipeline Automation - Provides automation for transitioning machine learning models from experimental development to production-ready scalable pipelines.
• State Checkpointing - Provides an integrated checkpointing system that persists intermediate data to allow resuming failed runs from the last successful step.
• Distributed Compute Clusters - Launches temporary groups of communicating cloud instances to support distributed algorithms and gang-scheduled workloads.
• Cloud Computing & Serverless - Distributes tasks across cloud CPUs and GPUs to handle massive parallel or gang-scheduled workloads.
• Workflow Execution Scaling - Scales the execution of data pipelines across distributed nodes and cloud clusters for high-availability production environments.
• Distributed Computing Frameworks - Distributes computational workloads across cloud CPUs and GPUs using ephemeral clusters and spot instances.
• Pipeline - Saves and restores intermediate state and artifacts between steps to enable resuming failed runs.
• Stateful Run Resumption - Restarts a pipeline from the last successful step to avoid re-running expensive computations.
• Workflow Orchestration - Structures pipeline execution as a directed acyclic graph of steps with support for conditional branching and parallel execution.
• State Persistence - Saves flow variables to a remote object store at step boundaries to allow execution resumption and data sharing.
• State Persistence - Stores objects and dataframes at step boundaries to enable result inspection and execution resumption.
• Distributed Training Managers - Integrates with specialized training tools to manage cluster formation and node communication for ML.
• Cloud Data Access - Connects to cloud object storage to retrieve and store large datasets efficiently.
• Pipeline Visualization - Renders data from workflow executions into visual cards for easier analysis of the data flow.
• Data Querying - Fetches input data from data warehouses using SQL queries or loads data files directly from storage.
• Data Warehouse Exporters - Writes predictions and computation outputs to data warehouses or caches to power downstream systems.
• Parallel Data Merging - Resolves and propagates data artifacts from multiple parallel branches into a single join step.
• Workflow Result Retrieval - Enables access to data and outputs from previous pipeline executions for analysis or reuse.
• Notebook Execution Environments - Runs defined workflows directly within a notebook cell and returns a run object for inspection.
• Metadata Querying - Retrieves and analyzes results across multiple experiments using a programmatic interface.
• Custom Parallel Task Execution - Executes multiple versions of a task concurrently where each instance runs independently.
• Executable Packaging - Snapshots local code and external dependencies into a portable package for consistent remote execution.
• Cloud Infrastructure Management - Manages remote execution environments and spot instances to optimize costs while scaling resources.
• Environment Management - Defines specific software libraries and packages for individual steps to ensure consistent execution environments.
• Dependency-Isolated - Packages specific software libraries and code snapshots into portable images to ensure consistent execution across remote compute nodes.
• Data Workflow Execution - Executes notebook-defined flows on cloud infrastructure instead of the local instance.
• Distributed Task Orchestration - Launches ephemeral compute clusters to run tasks that must communicate and coordinate during execution.
• Event-Driven Workflow Triggers - Starts workflows automatically in response to external signals or the completion of upstream dependent pipelines.
• ML Lifecycle Orchestration - Coordinates the full sequence of machine learning stages from local prototyping to production deployment.
• Production Deployment Tools - Transfers developed pipelines to high-availability orchestrators with support for reactive execution.
• Remote Execution Proxies - Routes local notebook-defined flows to remote cloud infrastructure for vertical scaling of memory and GPU resources.
• Spot Instance Orchestration - Reduces cloud costs by using interruptible instances and automatically retrying tasks before termination.
• Vertical Scaling Tools - Executes specific tasks on remote cloud instances to access more memory or GPU power.
• Workflow Run Management - Starts workflows in the background to monitor progress and stream logs without blocking the interface.
• Runtime Exception Handling - Catches exceptions during a task to prevent workflow termination and allow subsequent steps to continue.
• Failure Handling Policies - Manages task-level failures using retries and exception catching to ensure pipeline completion.
• Data-Driven Routing - Implements conditional branching that routes the pipeline execution based on the value of computed data artifacts.
• Complex Workflow Coordination - Coordinates dependent execution chains by linking multiple independent pipelines through an event-driven architecture.
• Task Retry Policies - Restarts a task automatically after a failure to resolve transient platform issues.
• Execution Run APIs - Extracts data values and metadata produced by specific tasks within previous pipeline executions.
• Machine Learning Operations - A human-friendly Python library for helping scientists and engineers build and manage real-life data science projects.
• Workflow Orchestration - Framework for building and managing end-to-end data science projects.
• Data Pipelines - Simplifies building and managing real-world data science projects.
• General Purpose Orchestration - Human-friendly library for building and managing data science projects.

Star history