airbnb/airflow

Features

• DAG-Based Dependency Resolution - Uses Directed Acyclic Graphs to determine the exact execution order and dependency mapping of complex workflows.
• Metadata Databases - Provides a centralized relational database to store the state, history, and configuration of all workflows and tasks.
• Data Orchestration Platforms - Serves as a comprehensive platform for authoring, scheduling, and monitoring the entire lifecycle of data processing workflows.
• Data Pipeline Orchestration - Provides a comprehensive system for defining, scheduling, and monitoring complex sequences of data processing tasks.
• Workflow Schedulers - Includes mechanisms to trigger workflows based on defined timetables, cron expressions, or specific external events.
• Distributed Processing - Offloads and distributes heavy computational workloads across a cluster of worker nodes for parallel processing.
• Distributed Task Queues - Implements a distributed task queue to execute workloads across multiple worker nodes via a message broker.
• Distributed Task Workers - Runs sequences of tasks across a cluster of distributed workers based on defined dependencies and schedules.
• Scheduled Job Execution - Automates the regular execution of Python scripts and tasks based on timetables or specific trigger events.
• Scheduling Loops - Employs a continuous loop that monitors the metadata database to trigger tasks based on time and dependency status.
• Task Execution Engines - Features an execution engine that manages background tasks using distributed processing models and persistent worker pools.
• Task Scheduling - Implements a scheduler that calculates task execution order based on Directed Acyclic Graph dependencies.
• Workflow Task Definitions - Enables the definition of task sequences and dependencies using Python code for maintainable and versionable pipelines.
• Parallel Historical Backfilling - Provides capabilities to trigger workflow executions for past date ranges to populate missing or updated historical data.
• Inter-Task Metadata Sharing - Facilitates the sharing of small data packets and metadata between tasks within a single workflow execution.
• Dynamic Task Graphs - Enables the construction and modification of execution paths at runtime using Python code to handle variable workloads.
• Metadata Exchange - Allows the exchange of small state fragments between tasks using key-value pairs stored in the central database.
• Automation Workflow Monitors - Offers a visual interface to track the success, timing, and failure states of automated data pipelines.
• Data Pipeline Monitoring - Provides a web-based monitor for tracking real-time status, throughput, and execution history of data integration jobs.
• Application Health Monitors - Includes a monitoring system to track the operational health and progress of running tasks to identify failures.
• Real-time Pipeline Progress Visualization - Provides a web interface for visualizing real-time execution progress and task status to troubleshoot data pipeline failures.
• Workflow Frameworks - Python-based system for orchestrating complex workflow dependencies.

Star history