trekhleb/homemade-machine-learning
Homemade Machine Learning
This project provides a collection of machine learning algorithms implemented from scratch in Python. It serves as an educational resource using interactive notebooks that combine code with mathematical explanations to demonstrate the first principles of data science.
The repository includes reference implementations for neural networks, such as multilayer perceptrons with backpropagation, and supervised learning models including linear and logistic regression. It also covers unsupervised learning through k-means clustering and Gaussian anomaly detection.
The codebase covers a broad range of mathematical foundations, including iterative gradient descent, closed-form normal equations, L2 regularization, and polynomial feature transformations. Supporting utilities are provided for feature scale normalization, performance metric calculation, and sample dataset provisioning.
Features
- Multilayer Perceptrons - Provides a from-scratch implementation of multilayer perceptrons with input, hidden, and output layers.
- Machine Learning Education - Serves as an educational resource for learning the mathematical foundations of ML through scratch implementations.
- Backpropagation - Implements the backpropagation algorithm to calculate gradients and update neural network weights.
- Classification - Implements multi-class classification using a one-vs-all approach to categorize inputs.
- K-Means Clustering - Provides a manual implementation of k-means clustering for unsupervised data partitioning.
- Logistic Regression Models - Implements logistic regression for binary classification by fitting data to a logistic curve.
- Machine Learning Implementations - Provides code-based reference implementations of core machine learning algorithms written from first principles in Python.
- Linear Regression Implementations - Provides an educational from-scratch implementation of linear regression using gradient descent.
- Gradient Optimization Techniques - Applies gradient optimization and regularization techniques to improve model convergence and accuracy.
- Forward Propagation Engines - Provides the computational logic for the forward pass in multilayer perceptrons.
- Neural Network Implementations - Builds multilayer perceptrons from scratch, focusing on the low-level mechanics of neural networks.
- Gradient Descent Algorithms - Implements gradient descent as the primary iterative optimization method for minimizing cost functions.
- Supervised Learning - Implements supervised learning workflows including linear and logistic regression for prediction and classification.
- Unsupervised Learning - Implements unsupervised learning workflows for clustering and anomaly detection in unlabeled data.
- Backpropagation Training - Implements the backpropagation algorithm from scratch to enable training of multilayer perceptrons.
- Model Prediction Logic - Provides feedforward propagation to identify the most probable class labels for input data.
- Anomaly Detection - Implements a Gaussian distribution-based system to identify outliers as low-probability observations.
- Feature Scale Normalization - Provides mean normalization utilities to ensure stable convergence during gradient descent training.
- Feature Expansion - Provides utilities for polynomial feature expansion to model non-linear relationships within linear regression.
- Polynomial Feature Mapping - Enables non-linear relationship modeling by transforming independent variables into polynomial features.
- Closed-Form Solvers - Implements a closed-form solver using the normal equation to compute optimal model parameters without iteration.
- Performance Metrics - Provides utilities to compute F1 score, precision, and recall for model performance measurement.
- Regularization Techniques - Implements L2 regularization to penalize large coefficients and prevent model overfitting.
- Clustering and Density Estimation - Implements unsupervised techniques including K-Means clustering and Gaussian density estimation for outlier detection.
- Weight Initialization - Includes utilities for initializing neural network weights to break symmetry during training.
- L2 Regularization - Includes L2 regularization (ridge penalty) to prevent overfitting by penalizing large coefficient magnitudes.
- Educational Notebooks - Provides interactive Jupyter notebooks that interleave Python code with mathematical explanations.
- Supervised Learning Examples - Offers practical examples of supervised learning, including linear/logistic regression and regularization.
- Data Science Projects - Implementations of machine learning algorithms from scratch.
- Educational Resources - Python implementations of algorithms with interactive demos.
