nteract/papermill
Papermill
Papermill is a Jupyter notebook execution engine and parameterization framework designed to run notebooks programmatically. It allows users to inject custom input values into notebooks to execute the same logic across different datasets, transforming interactive notebooks into reproducible data science pipelines.
The project functions as a language-agnostic notebook runner and orchestrator, supporting kernels for Python, R, Julia, and Scala. It is distinguished by its cloud-integrated runner capabilities, featuring built-in handlers to read and write notebooks directly to storage providers such as Amazon S3, Azure Blob Storage, and Google Cloud.
The system provides a comprehensive surface for automation and observability. This includes a command-line interface for triggering executions, API bindings for script integration, and tools to monitor execution progress and track state via incremental persistence.
Users can extend the framework by implementing custom execution engines and I/O handlers to support additional storage backends or runtime environments.
Features
- Notebook Parameterization - Enables the injection of custom input values into notebooks to run the same logic across different datasets.
- Notebook Parameterization Tools - Allows running notebooks with different input parameters to generate multiple output files for various datasets.
- Notebook Code Execution - Executes notebook files with specific input parameters and saves the resulting output to a designated path.
- Data Pipeline Automation - Integrates notebooks into scripts and workflows to process data and generate reports programmatically.
- Parameter Injection - Injects custom input values into notebooks by creating a parameter cell to run the same logic across different datasets.
- Parameter Injection Cells - Provides the ability to inject specialized parameter cells into notebooks to override default variables.
- Notebook Automation Tools - Automates notebook execution as part of larger scripts or workflows for data processing.
- Notebook Parameterization - Injects custom input values into notebooks so that the same logic executes with different data or settings.
- Notebook Execution Environments - Implements a backend to handle the execution of notebook cells across different environments or runtimes.
- Parameterized Execution - Runs notebooks using specific input parameters via API or CLI to generate new output files.
- Cloud-Integrated Notebook Runners - Ships built-in handlers to read and write notebooks directly to storage providers such as Amazon S3, Azure Blob Storage, and Google Cloud.
- Multi-Kernel Execution - Operates as a language-agnostic runner supporting kernels for Python, R, Julia, and Scala.
- Notebook Execution Engines - Functions as a programmatic execution engine for Jupyter notebooks to generate reproducible data science pipelines.
- Notebook Workflow Orchestrators - Provides the ability to trigger a sequence of notebooks to build complex data processing and analysis pipelines.
- Notebook Execution Engines - Provides a base class to implement custom notebook execution logic and integrate with different client processes.
- Language-Agnostic Runtimes - Supports notebook execution across multiple programming languages including Python, R, Julia, and Scala through a unified interface.
- File I/O Decoupling Patterns - Decouples notebook reading and writing from execution logic to support various local and cloud storage backends.
- Notebook Storage Backends - Manages the persistence of notebook files across Azure Blob Storage, Azure Data Lake, and Amazon S3.
- Programmatic Notebook Orchestrators - Triggers sequences of notebooks and saves executed outputs to local or cloud storage as part of a workflow.
- Notebook Workflow Orchestration - Triggers sequences of notebooks based on prior execution results to build complex data processing pipelines.
- Kernel Parameter Translators - Maps programming languages to translator objects to handle parameterization for various notebook kernels.
- Language Type Mappers - Translates generic parameter values into native objects compatible with the target notebook kernel's language.
- Library and CLI Interfaces - Provides both a programmatic library API and a command-line interface to trigger notebook executions.
- Managed Cloud Notebooks - Supports the storage and retrieval of notebooks across local filesystems and cloud providers.
- CLI Execution - Runs notebooks in a subprocess via a command-line interface, saving results to a file or stream.
- Notebook Integrations - Provides bridge utilities to trigger and manage notebook executions from within a programming language.
- Incremental State Persistence - Saves the notebook file to storage after every cell completion to preserve state during failures.
- Execution Auditing - Enables the execution and inspection of notebooks to track timing, status, and results for auditing.
- Remote Filesystem I/O - Provides mechanisms to read and write notebooks to remote cloud storage and local filesystems.
- Custom Storage Handlers - Allows developers to implement custom logic for reading and writing notebooks to external storage services.
- Execution Backend Abstractions - Implements a common interface to decouple notebook execution logic from specific language kernels and runtimes.
- Kernel Resolution - Inspects internal notebook metadata to automatically identify and launch the corresponding execution environment.
- General Utilities - Tool for parameterizing and executing Jupyter notebooks.
- Interactive Notebooks - Parameterizes and executes notebooks as automated scripts.
- Jupyter Ecosystem - Parameterizing and executing notebooks programmatically.
- Publishing and Conversion - Tool for parameterizing and executing notebooks for automated analysis.
- Dokumentation und Wissen - Tool for parameterizing and executing Jupyter Notebooks.
