apache/incubator-airflow

Features

• Data Pipeline Automation - Provides a comprehensive system for defining and executing structured data transformation and aggregation workflows.
• Data Pipeline Orchestration - Provides a platform for defining, scheduling, and monitoring complex sequences of data processing tasks and their dependencies.
• Workflow Scheduling And Orchestration - Triggering complex data processes based on specific time intervals or external events to ensure reliable execution.
• Data Pipeline Orchestrators - Acts as a data pipeline orchestrator managing execution sequences of dependent transformation tasks.
• Metadata and State Management - Manages the persistence of task status and execution history in a relational database to coordinate distributed state.
• Data Engineering Pipelines - Provides a framework for automated collection, transformation, and loading of data across diverse sources.
• Python Data Pipeline Frameworks - Provides a Python-based framework for building and managing complex batch data workflows and DAGs.
• Task Dependency Management - Implements a dependency-aware execution engine to ensure tasks run in the correct order based on a defined graph.
• Workflow Schedulers - Features a scheduler that triggers data pipeline executions based on cron expressions, time intervals, or external events.
• Task Schedulers - Includes a scheduler that periodically scans metadata to trigger persistent background tasks.
• Distributed Worker Orchestration - Provides a distributed orchestration environment to dispatch tasks across a fleet of remote worker nodes.
• Data Pipeline Definitions - Allows pipelines to be authored as Python code, making them maintainable and versionable through standard software engineering practices.
• DAG-Based Dependency Resolution - Implements a DAG engine to determine the precise execution order of interdependent pipeline tasks.
• Directed Acyclic Graph Engines - Implements a DAG engine to ensure precise execution order and failure recovery for modular nodes.
• Operator-Based Task Abstractions - Employs an extensible system of operators to encapsulate reusable integration logic for pipeline tasks.
• Programmatic Workflow Authoring - Allows authoring data pipelines as code to ensure they are maintainable, versionable, and testable.
• Pipeline Monitoring Dashboards - Ships a web-based dashboard for tracking the status and health of data pipelines.
• Workflow Management Dashboards - Offers a management interface for monitoring and interacting with pipeline execution environments.
• Custom Task Operator Extensions - Allows developers to extend pipeline capabilities by defining and registering custom task operators.
• Backfill Managers - Includes capabilities to execute workflows over past date ranges to recover missing data or correct historical records.
• Template-Based Logic Engines - Uses a template-based logic engine to inject runtime variables and dates into task definitions.
• Pipeline Definition Templating - Provides a templating system to apply dynamic parameters and customizations to pipeline definitions.
• Execution History Tracking - Tracks execution history and status of workflows to identify failures and ensure data reliability.
• Real-time Pipeline Progress Visualization - Provides a web interface to visualize real-time pipeline progress and troubleshoot execution issues.
• Scheduling - Programmatically authors and schedules complex data workflows.
• Workflow and Pipeline Automation - Workflow automation and scheduling system for data pipelines.
• Workflow Scheduling - Programmatically author, schedule, and monitor complex workflows.
• General Purpose Orchestration - Python-based platform for managing directed acyclic graphs of tasks.
• Workflow Scheduling - Platform for programmatically authoring, scheduling, and monitoring complex workflows.
• Example Applications - System for authoring and monitoring data pipelines.

Star history