quantumblacklabs/kedro

Features

• Data Pipeline Orchestration - Manages the execution order and dependencies of data processing tasks to move data from raw sources to models.
• Production Data Science Toolboxes - Provides a comprehensive set of tools to move data analysis from interactive notebooks into maintainable production pipelines.
• Data Science Frameworks - Provides a framework for building reproducible and modular data science workflows using software engineering best practices.
• Productionization Frameworks - Builds reproducible and maintainable data science workflows using software engineering patterns for environmental consistency.
• Model Versioning Systems - Tracks versions of datasets and models within a catalog to ensure the reproduction of specific results.
• Data Catalogs - Manages a central registry to track data versions and coordinate access across diverse storage systems.
• Data Connectors - Provides lightweight connectors for loading and saving data across diverse file formats and storage systems.
• Data Access Abstractions - Provides an abstraction layer that decouples data processing logic from the physical storage backends via a central registry.
• Data I/O - Implements a pluggable system of loaders and savers to handle diverse file formats and cloud storage systems.
• Graph-Based Workflow Orchestrators - Resolves function dependencies to execute data processing tasks in a sequence defined by a directed acyclic graph.
• Modular Data Pipelines - Organizes data engineering and science workflows into modular components to ensure maintainable and reproducible results across environments.
• Pipeline Component Modularization - Isolates data processing logic into independent, swappable functional units to separate business logic from the execution framework.
• Data Engineering Workflow Patterns - Structures the extraction and transformation of data into modular components for improved maintenance and reuse.
• Dependency-Aware Task Orchestration - Automatically resolves dependencies between processing functions to ensure data tasks are executed in the correct sequence.
• Workflow Visualizations - Generates visual maps of data pipelines to help users identify and analyze dependencies between processing steps.
• Data Dependency Visualizers - Provides visual representations of project workflows to map dependencies between different pipeline steps.
• Exploration to Production Bridges - Integrates Jupyter notebooks and exploratory analysis with structured production pipelines for a seamless development cycle.
• Notebook Integrations - Connects interactive Jupyter notebooks with structured pipelines to transition analysis from exploration to production.
• Project Scaffolding Templates - Applies modifiable templates based on engineering best practices to ensure consistent project organization.
• Workflow Execution Scaling - Executes data processing tasks across single machines, distributed clusters, or managed orchestration platforms.
• Environment Configuration - Manages project parameters through a layered configuration system that adapts to different deployment targets.
• Project Structures - Enforces standardized directory layouts and configuration systems to ensure consistency across data science environments.
• Logic Reusability Patterns - Decouples data logic from orchestration using software engineering patterns to enable cross-project code sharing.
• Workflow Orchestration - Implements software engineering best practices for ML pipelines.
• GitOps And Continuous Delivery - Python framework for creating reproducible and modular data science pipelines.
• Infrastructure and Serving - Build data pipelines.
• MLOps and Workflows - Framework for creating modular and reproducible data science code.
• Workflow Frameworks - Development tool for building modular data pipelines.

Star history