WenDesi/lihang_book_algorithm
Lihang Book Algorithm
This is an educational Python implementation of every algorithm from Li Hang's textbook on statistical learning methods. The project provides a comprehensive collection of supervised learning algorithms covering classification, regression, and sequence modeling techniques, implemented from scratch for learning and reference purposes.
The repository covers a broad range of foundational machine learning methods, including decision trees built using the ID3 algorithm with information gain, ensemble boosting through AdaBoost that combines threshold-based weak learners, and probabilistic sequence modeling with hidden Markov models using forward-backward, Baum-Welch, and Viterbi algorithms. It also includes maximum entropy and softmax regression models for multi-class classification, logistic regression for binary outcomes, naive Bayes classifiers for text and high-dimensional data, k-nearest neighbors classification by majority vote, binary perceptron classifiers, and support vector machines that find maximum-margin separating hyperplanes.
Additional capabilities include weight decay regularization to prevent overfitting, numerical techniques for handling high-dimensional probability computations, and a C++ extension that accelerates computationally intensive training loops. The implementations are designed to be runnable and testable, with the code organized to mirror the textbook's chapter structure for easy cross-referencing.
Features
- Statistical Learning Implementations - Provides complete Python implementations of every algorithm from Li Hang's statistical learning methods textbook.
- Decision Trees - Implements decision tree classifiers using the ID3 algorithm with information gain for classification.
- AdaBoost Classifiers - Ships an AdaBoost implementation that combines threshold-based weak learners into a strong binary classifier.
- Hidden Markov Models - Provides hidden Markov model implementations with forward-backward, Baum-Welch, and Viterbi algorithms.
- ID3 Algorithm Implementations - Implements the ID3 algorithm specifically for building decision trees using information gain.
- Softmax Classifiers - Trains a softmax regression model on digit images to classify handwritten digits into ten categories.
- K-Nearest Neighbor Classifiers - Implements K-nearest neighbors classification by majority vote from scratch for educational purposes.
- Logistic Regression Models - Implements logistic regression classifiers for binary outcome prediction using a sigmoid function.
- Naive Bayes Classifiers - Implements Naive Bayes classifiers using Bayes' theorem with strong feature independence assumptions.
- Perceptron Classifiers - Ships a binary perceptron classifier implementation trained on digit image data for pattern recognition.
- Support Vector Machines - Finds the optimal maximum-margin hyperplane that separates classes using support vectors.
- Softmax Regression - Applies multinomial logistic regression to assign probability distributions across multiple classes.
- Maximum Entropy Classifiers - Implements maximum entropy models alongside softmax regression for multi-class digit classification.
