ML Data and Artifact Versioning
For a version control system for ML data, the strongest matches are treeverse/dvc (DVC is a Git-integrated data versioning tool and pipeline), treeverse/lakefs (lakeFS is a data lake versioning system that provides) and iterative/dvc (DVC is the leading open-source data version control tool). attic-labs/noms and wandb/client round out the shortlist. Each is ranked by relevance to your query, popularity and recent activity.
Tools for tracking, versioning, and managing large machine learning datasets and model artifacts like code.
ML Data and Artifact Versioning
Find the best repos with AI.We'll search the best matching repositories with AI.
- treeverse/dvc
treeverse/dvc
15,679View on GitHubDVC is a data versioning tool and pipeline orchestrator designed to track large datasets and machine learning models using external storage and metadata pointers. It integrates with Git by utilizing placeholders to keep heavy artifacts out of the repository while maintaining a versioned link between code and data. The system manages remote data caches through a synchronization layer that connects local environments to cloud storage or network filesystems. It also functions as an experiment tracker, recording hyperparameters and metrics to compare the performance of different model iterations.
DVC is a Git-integrated data versioning tool and pipeline orchestrator that tracks datasets and ML models via external storage, provides experiment tracking with metrics comparison, and supports cloud storage sync — directly matching the request for Git-like version control of data and ML artifacts.
PythonGit-Integrated Data Versioning - treeverse/lakefs
treeverse/lakeFS
5,406View on GitHublakeFS is a data lake versioning system that provides Git-like branching and commits for large datasets stored in object storage. It functions as a version control layer, enabling the creation of immutable snapshots, atomic commits, and zero-copy branching to create isolated environments for data experimentation without duplicating physical files. The system serves as an S3-compatible storage gateway and an Iceberg REST catalog, allowing standard cloud storage protocols and compatible clients to manage versioned tables. It acts as a data quality gatekeeper by using an event-driven hook system
lakeFS is a data lake versioning system that provides Git-like branching and commits for large datasets stored in object storage, making it a direct fit for version controlling datasets with Git workflows; while it focuses on data lakes rather than ML-specific artifacts, it covers dataset versioning, snapshotting, and cloud integration.
GoDifferential Dataset ComparisonsData Versioning SystemsS3-Compatible Storage Adapters - iterative/dvc
iterative/dvc
15,680View on GitHubDVC is a data versioning tool and pipeline orchestrator designed to track large datasets and machine learning models. It functions as a system for managing large data artifacts by storing lightweight metadata in version control while keeping the actual binaries in a separate cache. The project serves as an experiment tracker and remote storage synchronizer, enabling the execution and comparison of machine learning iterations based on hyperparameters and performance metrics. It provides a bridge for pushing and pulling these large data artifacts between local environments and cloud or on-premi
DVC is the leading open-source data version control tool that tracks datasets and ML artifacts with Git-like workflows, supporting large file storage, pipeline orchestration, model registry, dataset snapshotting, cloud storage sync, and diff/compare—covering everything needed for this search.
PythonDataset Versioning SystemsPointer-Based TrackingContent-Addressable Storage - attic-labs/noms
attic-labs/noms
7,422View on GitHubNoms is a distributed version control database and content-addressable data store. It identifies data by cryptographic hashes to ensure integrity and deduplication, while tracking dataset state changes through a sequence of immutable commits to enable branching, forking, and historical recovery. The system functions as a peer-to-peer data synchronizer, reconciling state between disconnected database instances to ensure all nodes converge on the same data. It distinguishes itself as a schema-flexible document store that supports self-describing types, allowing schemas to evolve and widen as ne
Noms is a distributed version-control database that uses commits and branching to version datasets, directly matching the Git-like workflow you need for data versioning, though it does not include ML-specific pipeline tracking or a model registry.
GoVersioned Dataset Snapshots - wandb/client
wandb/client
11,128View on GitHubThis project is a collection of utilities designed for machine learning experiment tracking, data versioning, and the observability of large language model applications. It provides a client for recording hyperparameters and metrics during training to visualize performance trends and compare different model versions. The tool includes a model evaluation framework that uses custom scorers and automated judges to assess the quality of generated text outputs. It also provides observability tools to monitor and debug the execution flow and runtime behavior of language model applications. The sys
This repository is the Python client for Weights & Biases, a platform that provides Git-like versioning for datasets and ML artifacts, experiment tracking, model registry, and dataset snapshotting — exactly the kind of tool this search is after, though you'll need the wandb service to store and retrieve versions.
PythonExperiment TrackingArtifact VersioningData Lineage - git-lfs/git-lfs
git-lfs/git-lfs
14,336View on GitHubGit Large File Storage is a Git extension that replaces large binary assets with lightweight pointers to keep repository history fast and lean. It functions as a remote binary asset store, hosting large files on a separate server instead of storing them directly in the Git history. The system includes a binary file locking mechanism to prevent concurrent edits on large assets, ensuring only one user can modify a file at a time. It also provides a virtual file system mount to expose large assets as a local filesystem without requiring a full manual checkout, and a history migration tool to con
git-lfs is a Git extension for storing large binary files but lacks the ML-specific versioning, pipeline tracking, and model registry features that this search requires for data and ML artifact version control.
GoGit Large File StorageLarge File Storage (LFS) - takezoe/gitbucket
takezoe/gitbucket
9,371View on GitHubGitBucket is a self-hosted Git hosting platform and forge designed for managing private repositories. Built with the Scala language, it provides a web interface for version control and is implemented as a server compatible with the GitHub API to ensure integration with existing third-party tools. The platform allows for customization of the version control environment through a plugin-based extension model, enabling the installation of third-party plugins to add specialized features. Its capability surface covers software project management via integrated issue trackers, pull requests, and w
GitBucket is a self-hosted Git hosting platform with Git LFS support, which can technically store large files, but it lacks the ML-specific pipeline tracking, model registry, and dataset snapshotting that defines a data and ML artifact version control tool.
ScalaGit Large File StorageLarge File Storage (LFS) - gogs/gogs
gogs/gogs
47,606View on GitHubGogs is a self-hosted Git service and collaborative code hosting platform. It functions as a version control manager that allows users to store and manage source code on their own infrastructure using SSH, HTTP, and HTTPS protocols. The platform distinguishes itself through comprehensive mirroring capabilities, acting as a tool to synchronize and mirror repositories and wikis from external hosting providers to a local instance. It is designed for secure, containerized deployment, supporting non-root user configurations to meet strict security requirements. Beyond basic hosting, it provides a
Gogs is a self-hosted Git service for source code with Git LFS support, but it does not provide pipeline tracking, model registry, or dataset-specific snapshotting for data science and ML workflows.
GoGit Large File StorageLarge File Storage (LFS)Data Snapshotting - fivethirtyeight/data
fivethirtyeight/data
17,394View on GitHubThis repository serves as a public archive for the raw datasets and analytical code used to support journalistic reporting. It functions as a platform for reproducible research, providing the necessary materials for users to verify published findings and conduct independent statistical analysis. The collection utilizes a versioned storage model to track historical changes to both data and processing scripts. By organizing information into a structured directory hierarchy, the repository maps specific journalistic projects to their corresponding inputs and outputs, ensuring that the methodolog
This repository is a public archive of datasets and code for journalistic reporting, not a tool for version controlling your own datasets and ML artifacts; it demonstrates versioned data sharing but lacks the Git-like workflows, pipeline tracking, and model registry you need.
Jupyter NotebookVersion-Controlled Datasets - hbons/sparkleshare
hbons/SparkleShare
4,959View on GitHubSparkleShare is a self-hosted file sharing platform and cross-platform synchronization application. It functions as a private alternative to cloud storage, using Git repositories as the primary backend to version and synchronize files across Linux, macOS, and Windows devices. The system employs Git Large File Storage to manage large binary files, ensuring data integrity and performance. It allows users to maintain private file repositories on self-managed servers to ensure data privacy and control. The application provides automated directory mirroring by monitoring local filesystem changes
SparkleShare is a self-hosted Git-based file sync tool, not a dedicated data and ML artifact versioning system—it lacks pipeline tracking, model registry, and ML-specific workflows that this search targets.
C#Large File Storage (LFS) - charmbracelet/soft-serve
charmbracelet/soft-serve
6,638View on GitHubSoft Serve is a self-hosted Git server that authenticates users via SSH public keys and provides a terminal-based user interface for browsing repositories, files, and commits. It stores repository data and configuration in either SQLite or PostgreSQL, and supports role-based access control with four permission levels for managing repository visibility and write access. The server can be deployed via Docker or managed as a systemd service, and supports webhook notifications for push, collaborator, and branch or tag events to integrate with external automation workflows. It also enables server-
Soft Serve is a self-hosted Git server with a terminal UI and webhook support, but it does not provide dataset/ML-specific versioning features such as pipeline tracking, model registry, or dataset snapshotting that this search targets.
GoGit Large File Storage - mlflow/mlflow
mlflow/mlflow
26,554View on GitHubMLflow is an ML lifecycle platform focused on experiment tracking and model registry, not a Git-like version control system for datasets and artifacts — it lacks native large file storage, dataset snapshotting, and data diffing that this search targets.
PythonModel RegistriesModel Versioning
