# wendesi/lihang_book_algorithm

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/wendesi-lihang-book-algorithm).**

5,827 stars · 1,945 forks · Python

## Links

- GitHub: https://github.com/WenDesi/lihang_book_algorithm
- Homepage: http://blog.csdn.net/wds2006sdo/article/category/6314784
- awesome-repositories: https://awesome-repositories.com/repository/wendesi-lihang-book-algorithm.md

## Description

This is an educational Python implementation of every algorithm from Li Hang's textbook on statistical learning methods. The project provides a comprehensive collection of supervised learning algorithms covering classification, regression, and sequence modeling techniques, implemented from scratch for learning and reference purposes.

The repository covers a broad range of foundational machine learning methods, including decision trees built using the ID3 algorithm with information gain, ensemble boosting through AdaBoost that combines threshold-based weak learners, and probabilistic sequence modeling with hidden Markov models using forward-backward, Baum-Welch, and Viterbi algorithms. It also includes maximum entropy and softmax regression models for multi-class classification, logistic regression for binary outcomes, naive Bayes classifiers for text and high-dimensional data, k-nearest neighbors classification by majority vote, binary perceptron classifiers, and support vector machines that find maximum-margin separating hyperplanes.

Additional capabilities include weight decay regularization to prevent overfitting, numerical techniques for handling high-dimensional probability computations, and a C++ extension that accelerates computationally intensive training loops. The implementations are designed to be runnable and testable, with the code organized to mirror the textbook's chapter structure for easy cross-referencing.

## Tags

### Artificial Intelligence & ML

- [Statistical Learning Implementations](https://awesome-repositories.com/f/artificial-intelligence-ml/statistical-learning-implementations.md) — Provides complete Python implementations of every algorithm from Li Hang's statistical learning methods textbook.
- [Decision Trees](https://awesome-repositories.com/f/artificial-intelligence-ml/decision-trees.md) — Implements decision tree classifiers using the ID3 algorithm with information gain for classification.
- [AdaBoost Classifiers](https://awesome-repositories.com/f/artificial-intelligence-ml/ensemble-learning/adaboost-classifiers.md) — Ships an AdaBoost implementation that combines threshold-based weak learners into a strong binary classifier.
- [Hidden Markov Models](https://awesome-repositories.com/f/artificial-intelligence-ml/markov-state-transition-models/hidden-markov-models.md) — Provides hidden Markov model implementations with forward-backward, Baum-Welch, and Viterbi algorithms.
- [ID3 Algorithm Implementations](https://awesome-repositories.com/f/artificial-intelligence-ml/decision-trees/id3-algorithm-implementations.md) — Implements the ID3 algorithm specifically for building decision trees using information gain. ([source](http://blog.csdn.net/wds2006sdo/article/details/52849400))
- [Softmax Classifiers](https://awesome-repositories.com/f/artificial-intelligence-ml/image-classification/softmax-classifiers.md) — Trains a softmax regression model on digit images to classify handwritten digits into ten categories. ([source](http://blog.csdn.net/wds2006sdo/article/details/53699778))
- [K-Nearest Neighbor Classifiers](https://awesome-repositories.com/f/artificial-intelligence-ml/k-nearest-neighbor-classifiers.md) — Implements K-nearest neighbors classification by majority vote from scratch for educational purposes. ([source](http://blog.csdn.net/wds2006sdo/article/details/51933044))
- [Logistic Regression Models](https://awesome-repositories.com/f/artificial-intelligence-ml/logistic-regression-models.md) — Implements logistic regression classifiers for binary outcome prediction using a sigmoid function. ([source](https://cdn.jsdelivr.net/gh/wendesi/lihang_book_algorithm@master/README.md))
- [Naive Bayes Classifiers](https://awesome-repositories.com/f/artificial-intelligence-ml/naive-bayes-classifiers.md) — Implements Naive Bayes classifiers using Bayes' theorem with strong feature independence assumptions. ([source](https://cdn.jsdelivr.net/gh/wendesi/lihang_book_algorithm@master/README.md))
- [Perceptron Classifiers](https://awesome-repositories.com/f/artificial-intelligence-ml/perceptron-classifiers.md) — Ships a binary perceptron classifier implementation trained on digit image data for pattern recognition. ([source](https://cdn.jsdelivr.net/gh/wendesi/lihang_book_algorithm@master/README.md))
- [Support Vector Machines](https://awesome-repositories.com/f/artificial-intelligence-ml/support-vector-machines.md) — Finds the optimal maximum-margin hyperplane that separates classes using support vectors. ([source](https://cdn.jsdelivr.net/gh/wendesi/lihang_book_algorithm@master/README.md))

### Education & Learning Resources

- [Softmax Regression](https://awesome-repositories.com/f/education-learning-resources/educational-resources/systems-applied-computing/machine-learning-education/softmax-regression.md) — Applies multinomial logistic regression to assign probability distributions across multiple classes. ([source](https://cdn.jsdelivr.net/gh/wendesi/lihang_book_algorithm@master/README.md))
- [Maximum Entropy Classifiers](https://awesome-repositories.com/f/education-learning-resources/educational-resources/systems-applied-computing/machine-learning-education/softmax-regression/maximum-entropy-classifiers.md) — Implements maximum entropy models alongside softmax regression for multi-class digit classification.