astronomer/dag-factory
Dag Factory
Dag-factory is a framework for constructing and managing Apache Airflow data pipelines through declarative configuration files. By replacing manual procedural code with structured YAML definitions, it enables the programmatic generation of complex workflow structures, task dependencies, and execution schedules.
The project distinguishes itself by mapping configuration keys directly to Python class constructors and operators, allowing for the dynamic instantiation of objects and custom logic. It supports hierarchical configuration inheritance to standardize settings across environments and provides mechanisms for injecting Kubernetes pod specifications directly into task definitions to ensure isolated, scalable execution.
The framework covers the full pipeline lifecycle, including automated file discovery, task-level dynamic mapping for parallel processing, and the attachment of metadata for external system integration. It also includes command-line utilities for validating configurations, triggering executions, and managing environment migrations.
Features
- Data Pipelines and Orchestration - Provides a framework for defining and managing complex Apache Airflow data workflows through declarative configuration.
- Declarative Pipeline Construction - Constructs data pipelines by parsing configuration files, allowing users to define workflow structures without manual procedural code.
- Declarative Workflow Definitions - Enables declarative data engineering by defining workflow logic and transformations in YAML instead of procedural code.
- Workflow Scheduling And Orchestration - Enables scheduling recurring data workflows using cron expressions, time intervals, or data-driven triggers defined in configuration files.
- Declarative Configuration Files - Defines data pipeline structures and task dependencies using structured YAML files instead of procedural code.
- Declarative Workflow Definitions - Defines complex task dependencies and execution schedules through structured configuration files rather than imperative scripts.
- DAG Workflow Executions - Constructs and executes directed acyclic graphs of tasks programmatically at runtime based on configuration definitions.
- DAG Workflow Pipelines - Constructs data pipelines declaratively by parsing YAML configuration files to generate DAG structures.
- Workflow Execution Managers - Structures intricate dependencies and parallel execution paths using nested task groups to manage large-scale data processing.
- Custom Operator Definitions - Maps user-defined objects and custom operator classes to configuration entries for specialized pipeline logic.
- Workflow Task Generators - Generates dynamic workflow structures and task instances at runtime based on configuration inputs.
- Containerized Task Execution - Executes individual pipeline tasks as isolated containers within a cluster using pod specifications defined in configuration.
- Kubernetes Pod Provisioning - Injects Kubernetes pod specifications into task definitions to enable isolated and scalable containerized execution.
- YAML Configuration Files - Standardizes data workflow definitions across environments using human-readable YAML configuration files.
- Configuration Inheritance - Supports hierarchical inheritance of configuration settings to standardize pipeline parameters across different environments.
- Declarative Configuration Management - Enables declarative definition of infrastructure resources and pipeline parameters by mapping configuration keys to class constructors.
- Dynamic Object Instantiation - Maps configuration strings to Python class constructors to dynamically instantiate complex objects at runtime.
- CLI Pipeline Dispatchers - Provides command-line utilities for triggering pipeline executions, validating configurations, and managing environment migrations.
- Dynamic Task Spawning - Generates multiple task instances from single configuration entries to handle parallel processing requirements automatically.
- Dynamic Task Mappings - Creates multiple task instances from a single configuration entry to process varying data volumes or parallelize operations.
- Pipeline Lifecycle Managements - Manages the full lifecycle of data pipelines, including validation, organization, and deployment across environments.
