Airflow is a platform for programmatically authoring, scheduling, and monitoring complex data pipelines. It functions as a workflow automation engine that manages the lifecycle of recurring business processes by executing code-defined task dependencies. By representing workflows as directed acyclic graphs, the system ensures that task execution order and data flow are explicitly defined and reliably maintained across distributed computing environments. The platform distinguishes itself through a highly modular, provider-based architecture that decouples core orchestration logic from external
Arrow is a cross-language development platform for in-memory data. It provides a standardized, language-independent columnar memory format designed to accelerate analytical operations and improve memory efficiency on modern computing hardware. By utilizing a schema-driven approach, the framework enables the efficient organization of both flat and nested data structures. The project functions as an analytical data processing engine that facilitates high-performance computation directly on memory-resident datasets. It distinguishes itself through a zero-copy architecture, which allows multiple
Apache Beam is a distributed data pipeline framework and unified data processing model designed to handle both bounded batch data and unbounded real-time streams. It provides a system for building scalable, data-parallel workflows that operate across compute clusters using a single programming model. The framework utilizes a cross-runner pipeline abstraction that decouples the data processing logic from the underlying execution backend, allowing the same pipeline to run on different distributed compute engines. It supports multi-language pipeline development by translating high-level code fro
Cookbook is a comprehensive knowledge base and reference repository for data engineering. It serves as a centralized directory for data architecture patterns, professional career roadmaps, and a curated collection of public datasets. The project provides a structured guide for transitioning into specialized data engineering roles through skill-matrix mapping and technical interview preparation. It further distinguishes itself by documenting real-world industry case studies and decomposing large-scale industrial implementations into repeatable architectural patterns. The repository covers a b
OpenLineage is an Open standard for metadata and lineage collection designed to instrument jobs as they are running. It defines a generic model of run, job, and dataset entities identified using consistent naming strategies. The core lineage model is extensible by defining specific facets to…
The main features of openlineage/openlineage are: Data Engineering.
Open-source alternatives to openlineage/openlineage include: apache/airflow — Airflow is a platform for programmatically authoring, scheduling, and monitoring complex data pipelines. It functions… apache/arrow — Arrow is a cross-language development platform for in-memory data. It provides a standardized, language-independent… apache/beam — Apache Beam is a distributed data pipeline framework and unified data processing model designed to handle both bounded… apache/calcite — Calcite is a framework for parsing, optimizing, and translating SQL queries into relational algebra for execution… apache/cassandra — Cassandra is a distributed NoSQL database and wide-column store designed for high availability and linear scalability.… andkret/cookbook — Cookbook is a comprehensive knowledge base and reference repository for data engineering. It serves as a centralized…