Awesome GitHub RepositoriesContainerized Pipelines

Pachyderm is a containerized, versioned, and lineage-tracked data pipeline platform that runs natively on Kubernetes. It combines a distributed file system backend with immutable data versioning, so every commit to a data repository creates an auditable snapshot, and every pipeline step executes as an isolated container. The platform is defined by a data-centric pipeline model where pipelines are specified by their input and output data repositories rather than explicit task sequences, and provenance is recorded as a directed acyclic graph of commits linking output data to its input sources an

Maestro is a distributed job scheduler and containerized data pipeline tool designed to coordinate complex sequences of tasks. It functions as a Kubernetes workflow orchestrator and MLOps automation platform, utilizing directed acyclic graphs to manage task dependencies and execution order across computing clusters. The system distinguishes itself through the use of isolated container environments for each workflow step, ensuring consistent runtime dependencies. It incorporates an asynchronous event bus to coordinate state transitions and provides lifecycle hook integration that dispatches sy