drivendataorg/cookiecutter-data-science
Cookiecutter Data Science
This project provides a standardized project directory structure and boilerplate templates for organizing data analysis and machine learning workflows. It serves as a reproducible analysis framework and workspace boilerplate designed to ensure consistency across data science projects.
The template distinguishes between exploratory research in notebooks and reusable, testable logic in modular Python packages. It enforces a convention-based directory hierarchy that treats the analysis pipeline as a directed acyclic graph by separating raw, interim, and processed data.
The framework covers a broad range of organizational needs, including machine learning asset management for model artifacts, secret isolation via environment variables, and task automation through makefiles. It also includes scaffolding for documentation, testing frameworks, linting, and environment configuration.
The project structure is generated using a command-line tool and supports the selection of specific template versions via git branches or tags.
Features
- Project Scaffolding - Generates a standardized directory hierarchy and boilerplate files to bootstrap consistent data science projects.
- Machine Learning Scaffolds - Provides a standardized boilerplate for organizing machine learning project directories, data stages, and model artifacts.
- Data Stage Categorization - Separates raw, interim, and processed data into distinct folders to maintain a clear lineage from source to final set.
- Filesystem Hierarchies - Provides a standardized directory structure to ensure data lineage and a reproducible analysis pipeline.
- Data Science Project Templates - Provides a standardized project directory structure for organizing data analysis and machine learning workflows.
- Workspace Generators - Generates a complete project workspace layout from templates, including makefiles and modular package structures.
- Interactive Data Science Environments - Bootstraps a professional workspace with integrated support for linting, testing, and environment management.
- Analysis Workspace Organization - Provides a structured layout for organizing raw data files, extraction scripts, and database credentials.
- Immutable Data Patterns - Treats the analysis pipeline as a directed acyclic graph by utilizing immutable data flow patterns.
- Modular Package Systems - Organizes analysis logic into a formal Python package to facilitate code reuse across notebooks and scripts.
- Project Bootstrapping Tools - Provides CLI-driven bootstrapping to generate a standardized and reproducible data science workspace.
- Project Organization - Establishes a predefined directory hierarchy to separate raw data, processed sets, and model artifacts.
- Project Structures - Enforces a consistent directory hierarchy to maintain a clear lineage from raw data to final analysis outputs.
- Analytical Reproducibility - Ensures analytical reproducibility by isolating raw data from processed sets and exploratory work from production code.
- Exploration and Production Separation - Enforces a strict separation between exploratory notebooks and testable, portable source code.
- ML Asset Management - Organizes a project hierarchy to track data lineage and store serialized models and intermediate artifacts.
- Model Management - Creates dedicated storage locations for serialized models, predictions, and model summaries.
- CLI Task Runners - Implements a CLI-based task runner for orchestrating environment setup and data management.
- Environment Bootstrapping - Provides a foundation for replicating the computational environment through package manager and VM integration.
- Environment Configuration Management - Generates configuration files for environment managers to handle dependency locking and package installation.
- Environment Variable Loaders - Includes mechanisms to load sensitive credentials from external environment files to keep secrets out of version control.
- Exploration to Production Bridges - Provides a structural bridge between rapid iteration in notebooks and reusable, production-ready source code.
- Task Automation Tools - Supplies a centralized makefile to automate repetitive development tasks and environment synchronization.
- Source Code Modularization - Encourages wrapping analysis logic into formal Python packages to allow shared code to be imported into notebooks.
- Research and Analysis Workflows - Segments project directories for notebooks, reports, and figures to organize the end-to-end research analysis workflow.
- Secrets and Credential Management - Isolates sensitive credentials from version control using environment files and automated loader mechanisms.
- Data Engineering Workflow Patterns - Separates data ingestion, feature engineering, and model training into modular Python components.
- Data Science Environments - Standardizes project structures for data science.
