Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
OpenAddresses is an open-source geospatial data aggregator and directory that collects public domain and open-license address, parcel, and building datasets from governments and organizations worldwide. It functions as a global index and data warehouse for locating and distributing free geospatial records. The project operates a normalization pipeline that cleans and standardizes diverse source formats into a consistent global coordinate and attribute schema. This process includes a crowdsourced curation pipeline and programmatic quality validation to verify the spatial accuracy and formattin
Airflow is a platform for programmatically authoring, scheduling, and monitoring complex data pipelines. It functions as a workflow automation engine that manages the lifecycle of recurring business processes by executing code-defined task dependencies. By representing workflows as directed acyclic graphs, the system ensures that task execution order and data flow are explicitly defined and reliably maintained across distributed computing environments. The platform distinguishes itself through a highly modular, provider-based architecture that decouples core orchestration logic from external
DBND an open source framework for building and tracking data pipelines. DBND is used for processes ranging from data ingestion, preparation, machine learning model training and production.
The main features of databand-ai/dbnd are: Data Pipelines, Open Source Catalogs.
Open-source alternatives to databand-ai/dbnd include: dagworks-inc/hamilton — Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode… openaddresses/openaddresses — OpenAddresses is an open-source geospatial data aggregator and directory that collects public domain and open-license… analysiscenter/batchflow. apache/beam — Apache Beam is a distributed data pipeline framework and unified data processing model designed to handle both bounded… apache/flume — Apache Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving… apache/airflow — Airflow is a platform for programmatically authoring, scheduling, and monitoring complex data pipelines. It functions…