kmkolasinski/deep-learning-notes
Deep Learning Notes
This repository is an educational collection of implementations and research notes focused on deep learning architectures and optimization techniques. It provides modular code examples designed to demonstrate foundational and advanced concepts in machine learning, ranging from basic neural network structures to complex training strategies.
The project distinguishes itself by offering practical implementations of specialized research methods, including capsule-based feature aggregation, gradient direction decoupling, and self-normalizing weight regularization. These materials allow for the study of how specific architectural choices and normalization strategies influence the stability and convergence of neural models.
The content covers a broad range of technical capabilities, including data pipeline balancing through resampling, computer vision tasks such as camera motion estimation, and the optimization of physical models. It also explores fundamental mathematical properties of gradient-based methods through the optimization of quadratic functions and the computation of optimal transport matrices. The repository consists of a series of Jupyter Notebooks that serve as a resource for academic learning and research experimentation.
Features
- Deep Learning Architectures - Provides modular code implementations and research notes for studying foundational and advanced neural network architectures.
- Neural Architecture and Training - Provides educational implementations of multilayer perceptrons and convolutional architectures for training and evaluation.
- Neural Network Implementations - Provides educational implementations of foundational and advanced neural architectures including autoencoders and neural ODEs.
- Computer Vision Features - Develops spatial pooling and capsule-based aggregation methods to improve visual feature representation.
- Sinkhorn Solvers - Calculates optimal transport between distributions using iterative Sinkhorn normalization of cost matrices.
- Feature Map Aggregators - Implements capsule-based feature aggregation to preserve spatial relationships and translational invariance in neural models.
- Machine Learning Resampling - Provides programmatic resampling techniques to balance minority class representation in training data pipelines.
- Machine Learning Optimization - Conducts experiments on the convergence properties of gradient-based methods and adaptive learning rates.
- Modular Architectures - Structures deep learning concepts into modular, isolated implementations for educational demonstration.
- Capsule Networks - Implements capsule-based pooling layers to explore alternative feature aggregation methods.
- Capsule Pooling Layers - Implements capsule-based pooling layers to aggregate feature map regions while preserving spatial relationships and translational invariance.
- Neural Network Stability Optimizations - Implements gradient normalization and weight regularization to improve the stability and convergence of neural models.
- Gradient Normalizers - Normalizes gradient vectors to decouple magnitude from direction, improving training stability.
- Weight Normalization - Implements weight normalization techniques to maintain statistical distributions and prevent vanishing gradients.
- Training Dataset Balancing - Balances training datasets by resampling minority classes to prevent bias during model learning.
- Weight Regularization - Regularizes neural weights to maintain self-normalizing properties in deep network layers.
- Training Data Pipelines - Implements data pipeline resampling to duplicate minority class samples for balanced training.
- Spatial Pooling - Performs dynamic feature pooling to preserve spatial representation and translational invariance.
- Deep Learning Education - Offers educational materials and implementations to help students understand neural network architectures and training techniques.
- Neural Network Implementations - Provides code examples demonstrating advanced training strategies including gradient normalization and capsule pooling.
- Deep Neural Network Training Optimization - Explores training stability through experiments with gradient normalization, weight regularization, and adaptive learning rates.
- Camera Trajectory Estimation - Analyzes sequential image frames to estimate camera trajectory and orientation in 3D environments.
- Optimal Transport Plan Computations - Computes optimal transport plans between data distributions through iterative matrix normalization.
- Physical Model Optimizers - Automates parameter scaling for physical tight-binding models to ensure design constraint satisfaction.
