pachyderm/pachyderm
Pachyderm
Pachyderm is a containerized, versioned, and lineage-tracked data pipeline platform that runs natively on Kubernetes. It combines a distributed file system backend with immutable data versioning, so every commit to a data repository creates an auditable snapshot, and every pipeline step executes as an isolated container. The platform is defined by a data-centric pipeline model where pipelines are specified by their input and output data repositories rather than explicit task sequences, and provenance is recorded as a directed acyclic graph of commits linking output data to its input sources and pipeline versions.
The platform differentiates itself through its Kubernetes-native orchestration, which manages pipeline execution as Kubernetes jobs for autoscaling, resource isolation, and cluster-wide scheduling. Trigger-driven pipeline invocation automatically starts downstream runs when new data commits arrive, enabling fully automated multi-stage data processing. Pachyderm also provides built-in data lineage tracking that records an immutable history of data versions and the pipeline steps that produced each version, supporting full reproducibility and auditability.
For deployment, Pachyderm supports both local cluster deployment using containers for development and testing, and production cluster deployment on Kubernetes in cloud environments. This allows teams to spin up a complete data pipeline environment on a local machine for iteration before provisioning a scalable, enterprise-grade data platform on cloud Kubernetes.
Features
- Data Lineage Trackers - Records an auditable history of data versions and the pipeline steps that produced each version.
- Data Pipeline Automation - Triggers multi-stage, language-agnostic data pipelines to run automatically whenever new data arrives.
- Repository-Defined Pipelines - Pipelines are defined by their input and output data repositories, not by explicit task sequences.
- Containerized Pipelines - Runs language-agnostic data pipelines inside containers for reproducible and portable data transformations.
- Pipeline-Integrated Versioning - Automates multi-stage data pipelines with built-in version control and lineage tracking for every dataset and transformation.
- Kubernetes-Native Data Platforms - Provisioning a complete, scalable data pipeline infrastructure on cloud Kubernetes for enterprise-grade data processing workloads.
- Immutable Commit Versioning - Every commit to a data repository creates an immutable snapshot, enabling full reproducibility and lineage tracking.
- Pipeline Step Execution - Each pipeline step runs as an isolated container, allowing language-agnostic processing and dependency encapsulation.
- Data Commit Triggers - New data commits automatically trigger downstream pipeline runs through a commit-based event system.
- Data Pipeline Platforms - Provisions a complete data pipeline platform on a Kubernetes cluster in a cloud environment.
- Data Processing Orchestrators - Deploying and scaling data pipelines on Kubernetes clusters, with automatic parallelization and resource management.
- Pipeline Job Orchestration - Orchestrates parallel, autoscaling pipeline execution across a Kubernetes cluster.
- Pipeline Job Orchestration - Pipeline execution is managed as Kubernetes jobs, enabling autoscaling, resource isolation, and cluster-wide scheduling.
- Distributed File Systems - Data is stored and versioned on a distributed file system (PFS), providing scalable, fault-tolerant storage for large datasets.
- Data Pipeline Orchestrators - Orchestrates containerized data pipelines on Kubernetes with automatic scaling and resource management.
- Data Provenance Commit Graphs - Data lineage is recorded as a directed acyclic graph of commits, linking output data to its input sources and pipeline versions.
- Immutable Lineage Trackers - Tracking and managing versions of datasets alongside the pipeline code that produced them, with full lineage and immutability.
- Immutable Data Lineage - Records an immutable history of data versions and the pipeline steps that produced each version.
- Data Pipeline Clusters - Ships a self-hosted data pipeline cluster deployable on local or cloud Kubernetes environments.
- Local Development Clusters - Spinning up a containerized data pipeline environment on a local machine for testing and iteration before production deployment.
- Data Management - Versions data using a Git-like approach for pipelines.
- Data Pipelines - Provides distributed processing for production ML pipelines on Kubernetes.
- Data Processing Tools - Platform for reproducible data science and data versioning.
- Build and CI/CD - Data versioning and pipeline management for containers.
Star history
How this analysis was created: This summary and feature list were written by an AI model that read the project's README and public documentation pages. Each feature links to the documentation it came from; stars, license and language come straight from the GitHub API. The model does not read the source code, and the analysis is refreshed when the project is re-analysed. Learn more on our About page.
