treeverse/dvc
Dvc
DVC is a data versioning tool and pipeline orchestrator designed to track large datasets and machine learning models using external storage and metadata pointers. It integrates with Git by utilizing placeholders to keep heavy artifacts out of the repository while maintaining a versioned link between code and data.
The system manages remote data caches through a synchronization layer that connects local environments to cloud storage or network filesystems. It also functions as an experiment tracker, recording hyperparameters and metrics to compare the performance of different model iterations.
The framework supports the definition of reproducible computational graphs by managing dependencies between code and commands. This capability enables the tracking of model lineage and the validation of data versioning consistency through commit-stage hooks.
Features
- Data Pipeline Orchestration - Orchestrates complex sequences of data processing tasks by managing dependencies between code and data.
- Model Lineage Trackers - Maintains a consistent provenance link between specific data versions, code, and hyperparameters used to produce a model.
- Machine Learning Experiment Trackers - Monitors metrics and hyperparameters across multiple model iterations to identify optimal performance.
- Data-Code Version Linking - Links specific versions of large datasets and models to the exact git commit of the code that produced them.
- Content-Addressable Storage - Implements content-addressable storage using cryptographic hashes to ensure data integrity and deduplicate large artifacts.
- Remote Dataset Caching - Provides a synchronization layer to cache large remote datasets locally using hash-based integrity verification.
- Dataset Versioning Platforms - Provides a workflow for tracking historical versions of large-scale datasets to ensure machine learning reproducibility.
- Artifact Versioning - Tracks large datasets and machine learning models using external caches and repository placeholders.
- Git-Integrated Data Versioning - Integrates large file tracking with Git by using placeholders to keep heavy artifacts out of the repository.
- Pointer-Based Tracking - Uses lightweight pointer files in Git to track large binary assets stored in an external cache.
- Data Pipeline Definitions - Allows users to define data processing pipelines as version-controlled code to ensure reproducibility.
- Directed Acyclic Graph Engines - Provides a DAG-based execution engine to manage computational dependencies between data and code.
- ML Pipeline Reproducibility - Defines dependencies between data and code to ensure computational graphs are rebuilt reliably across environments.
- Experiment Tracking - Records hyperparameters and metrics to compare the performance of different model iterations and training workflows.
- Cloud Storage Sync Tools - Synchronizes local data caches with remote cloud storage providers using standard transfer protocols.
- Backup Storage Backends - Provides drivers and configurations to offload large data caches to cloud or network storage providers.
- Experiment Result Comparators - Records hyperparameters and performance metrics in structured files to enable comparative analysis of model iterations.
