fmind/mlops-python-package
Mlops Python Package
The mlops-python-package serves as a standardized Python project template, data access abstraction layer, and workflow orchestrator for machine learning operations. It structures artificial intelligence workflows by connecting data processing and model training steps using directed acyclic graphs to manage execution order and inter-step dependencies.
The framework manages external configuration files and global service contexts to control program execution parameters and share dependencies across the application lifecycle. It includes strict type schema validation for tabular data frames and configuration inputs, alongside abstract dataset tagging that decouples storage logic from code by mapping abstract references to concrete readers and writers via configuration settings.
Additional capabilities cover automated hyperparameter optimization, machine learning experiment tracking to record parameters, metrics, and artifacts across training runs, and deterministic dataset splitting strategies for model training and evaluation. The package also supports specialized machine learning workflows including bikes availability forecasting, tuning, training, promotion, inference, evaluation, and explanation executed through the command line.
Features
- Directed Acyclic Graph Pipelines - Coordinates execution order and dependencies between individual processing steps by chaining tasks into typed dependency graphs.
- Dataset Abstraction Layers - Provides a data access abstraction layer that decouples storage logic from code using abstract configuration tags.
- Python Data Pipeline Frameworks - Serves as a workflow orchestrator that connects data processing and AI training steps using directed acyclic graphs.
- Strict Schema Enforcers - Enforces structural rules and field definitions on tabular data frames and configuration inputs at application startup.
- External Configuration Loading - Modifies program execution parameters and defines data inputs using external settings files to control pipeline stages without changing source code.
- Global Application Contexts - Maintains a shared runtime state and manages cross-cutting dependency injection across the entire application lifecycle.
- MLOps Templates - Provides a standardized Python project template for building machine learning pipelines, tracking experiments, and managing data workflows.
- YAML Configuration Files - Controls program behavior, pipeline stages, and runtime parameters by parsing external text files without modifying source code.
- Data Schema Validation - Enforces field definitions and validation constraints on tabular data structures to guarantee expected schemas.
- Storage Abstractions - Decouples data access from hardcoded paths by mapping abstract storage references in code to concrete readers and writers via configuration files.
- Machine Learning Experiment Trackers - Records parameters, metrics, and generated artifacts during training runs to compare performance and reproduce past iterations.
- Experiment Tracking - Records model parameters, metrics, and artifacts during training runs to compare performance and reproduce results across iterations.
- Hyperparameter Optimization - Scans predefined search spaces to discover top-performing model parameters through systematic automated experimentation strategies.
- Training and Testing Splits - Divides data into exclusive training, validation, and testing subsets using deterministic strategies to fit models and evaluate performance.
- DAG Workflow Pipelines - Coordinates pipeline steps using directed acyclic graphs to manage execution order and workflow dependencies.
- Machine Learning Pipelines - Executes specialized artificial intelligence workflows for tuning, training, promotion, inference, evaluation, and explanation via command line.
- Configuration Strictness Validators - Validates configuration files at startup using strict type definitions to prevent unexpected runtime execution errors.
