dagster-io/dagster

Features

• Data Pipeline Orchestration - Builds and schedules complex data workflows as version-controlled code to ensure reliable execution.
• Workflow Orchestration Engines - Coordinates distributed tasks and data dependencies across heterogeneous cloud environments and external infrastructure.
• Declarative Orchestration - Models data pipelines as a graph of versioned assets where the system automatically determines execution order based on dependency requirements.
• Data Lineage - Provides visual and searchable mapping of data flows to document relationships between source inputs and final downstream assets.
• Data Asset Lifecycle Management - Models data assets as first-class primitives to manage lineage, dependencies, and quality from ingestion through delivery.
• Data Asset Modeling - Models data as declarative assets to track lineage, quality, and dependencies throughout the entire lifecycle.
• Data Pipeline Definitions - Expresses data workflows, resources, and scheduling logic as version-controlled code to enable standard software engineering practices.
• Configuration-as-Code Frameworks - Expresses entire data workflows and infrastructure resources as version-controlled code to enable standard software engineering practices.
• Observability Platforms - Monitors the health, performance, and lineage of data workflows and assets through a unified interface.
• Data Ingestion - Provides modular tools to ingest and transform raw information from various sources into usable assets for downstream analysis.
• Data Partitioning - Divides large datasets into logical slices to enable incremental processing, targeted re-runs, and efficient management of high-volume workflows.
• Data Quality Frameworks - Integrates validation checks and automated policies directly into pipelines to ensure data integrity throughout the asset lifecycle.
• Data Catalogs - Maintains a centralized view of data assets, workflows, and lineage to help teams discover and reuse components.
• Orchestration Engines - Decouples the orchestration logic from the compute layer to allow tasks to run across diverse environments like containers or serverless.
• Managed Control Planes - Provides a hosted control plane with enterprise-grade features and insights for teams requiring fully managed infrastructure.
• Infrastructure as Code - Defines deployment environments and infrastructure configurations using declarative files to integrate with existing workflows.
• Infrastructure as Code Practices - Manages data pipelines and infrastructure configurations using software engineering practices like testing and CI/CD.
• Data Access Governance - Centralizes visibility across teams and environments while enforcing fine-grained access controls and maintaining audit-ready lineage for compliance.
• Automated Quality Workflows - Embeds validation checks and automated testing directly into data pipelines to ensure data integrity.
• Pipeline Observability Tools - Streams execution logs and performance metrics to external monitoring platforms to maintain unified observability and system health.
• Data Analysis and Processing - Orchestration for data assets and pipelines.
• Data Pipelines - Orchestrates data pipelines for machine learning and ETL.
• Data Pipelines and Orchestration - Orchestration platform for developing and observing data assets.
• Data Processing - Data orchestrator for ML and ETL workflows.
• Workflow Orchestration - Library for building and orchestrating data-intensive applications.
• Data Engineering - Data orchestrator for machine learning and ETL workflows.
• Job Schedulers - Orchestration platform for data assets.
• Scheduling - Orchestrates data pipelines for analytics and machine learning.
• Workflow Scheduling - Data orchestrator for machine learning and ETL pipelines.
• General Purpose Orchestration - Data orchestrator for machine learning, analytics, and ETL pipelines.
• Workflow Frameworks - API for defining DAGs to build data-intensive applications.
• Workflow Scheduling - Data orchestrator designed for machine learning, analytics, and ETL pipelines.
• Data Storage - Connects to cloud object stores, data warehouses, and lakehouse architectures to read, write, and version data assets securely.
• Event-Driven Data Pipelines - Executes data tasks based on custom schedules or external events to automate pipeline runs in response to real-time data changes.
• Distributed Computing - Launches and manages code execution across cloud-native, serverless, and containerized infrastructure to scale processing power.
• Deployment Environments - Supports self-hosted, hybrid, and managed infrastructure deployment models to avoid vendor lock-in.
• Resource Cost Management - Tracks and attributes resource consumption and compute expenses to specific data assets and pipeline runs.
• Secret Management - Retrieves and rotates sensitive API keys and configuration parameters from secure vaults automatically during pipeline execution.
• Execution Metadata - Captures and indexes execution logs, lineage, and asset state in a centralized store to provide unified visibility.
• Automated Alerting Workflows - Notifies teams immediately when data quality checks fail or budget thresholds are exceeded.
• Unit Testing - Verifies the logic and reliability of data processing code using unit, integration, and mock testing frameworks before deployment.
• Isolated Execution Environments - Creates ephemeral, production-mirroring sandboxes for pull requests to validate changes end-to-end.
• Local Development Tools - Provides isolated interfaces and testing frameworks that allow developers to simulate production environments and validate pipeline logic locally.
• Production Operationalization - Converts interactive notebooks and shell scripts into production-ready pipeline components to eliminate manual deployment overhead.
• Cloud Storage - Reads and writes data objects to cloud storage buckets to provide a resilient persistence layer for pipeline execution.
• Audit Logging - Maintains comprehensive logs of user actions and system changes to ensure compliance and visibility into operational history.
• Data Workflow Integrations - Incorporates conversational models and lifecycle management tools directly into data workflows to automate complex processing.
• Local Development Environments - Provides isolated interfaces for building, debugging, and mocking data workflows locally before production deployment.
• Data Synchronization and Consistency - Ensures reliable and uniform data outputs across different sources and timing intervals to maintain a single source of truth.
• User Identity Management - Integrates with enterprise identity providers to enforce role-based permissions and automate user provisioning through standard authentication protocols.
• Legacy Migration Strategies - Converts existing workflow definitions into modern data assets while maintaining backward compatibility during system transitions.

Star history