Lihang Book Algorithm

This is an educational Python implementation of every algorithm from Li Hang's textbook on statistical learning methods. The project provides a comprehensive collection of supervised learning algorithms covering classification, regression, and sequence modeling techniques, implemented from scratch for learning and reference purposes.

The repository covers a broad range of foundational machine learning methods, including decision trees built using the ID3 algorithm with information gain, ensemble boosting through AdaBoost that combines threshold-based weak learners, and probabilistic sequence modeling with hidden Markov models using forward-backward, Baum-Welch, and Viterbi algorithms. It also includes maximum entropy and softmax regression models for multi-class classification, logistic regression for binary outcomes, naive Bayes classifiers for text and high-dimensional data, k-nearest neighbors classification by majority vote, binary perceptron classifiers, and support vector machines that find maximum-margin separating hyperplanes.

Additional capabilities include weight decay regularization to prevent overfitting, numerical techniques for handling high-dimensional probability computations, and a C++ extension that accelerates computationally intensive training loops. The implementations are designed to be runnable and testable, with the code organized to mirror the textbook's chapter structure for easy cross-referencing.

Features

Statistical Learning Implementations - Provides complete Python implementations of every algorithm from Li Hang's statistical learning methods textbook.
Decision Trees - Implements decision tree classifiers using the ID3 algorithm with information gain for classification.
AdaBoost Classifiers - Ships an AdaBoost implementation that combines threshold-based weak learners into a strong binary classifier.
Hidden Markov Models - Provides hidden Markov model implementations with forward-backward, Baum-Welch, and Viterbi algorithms.
ID3 Algorithm Implementations - Implements the ID3 algorithm specifically for building decision trees using information gain.
Softmax Classifiers - Trains a softmax regression model on digit images to classify handwritten digits into ten categories.
K-Nearest Neighbor Classifiers - Implements K-nearest neighbors classification by majority vote from scratch for educational purposes.
Logistic Regression Models - Implements logistic regression classifiers for binary outcome prediction using a sigmoid function.
Naive Bayes Classifiers - Implements Naive Bayes classifiers using Bayes' theorem with strong feature independence assumptions.
Perceptron Classifiers - Ships a binary perceptron classifier implementation trained on digit image data for pattern recognition.
Support Vector Machines - Finds the optimal maximum-margin hyperplane that separates classes using support vectors.
Softmax Regression - Applies multinomial logistic regression to assign probability distributions across multiple classes.
Maximum Entropy Classifiers - Implements maximum entropy models alongside softmax regression for multi-class digit classification.

rasbt/python-machine-learning-book

12,614View on GitHub

This project is an educational resource providing practical code examples and implementations of machine learning algorithms using the Python language. It serves as a guide for constructing predictive pipelines, clustering models, and dimensionality reduction within the Scikit-Learn ecosystem. The repository includes comprehensive demonstrations for supervised and unsupervised learning, as well as detailed examples for implementing neural networks and deep architectures. It also provides practical guidance on exporting model parameters to JSON and wrapping trained models in web APIs for produ

biolab/orange3

5,635View on GitHub

Orange3 is a visual data mining platform that provides an interactive canvas for building data analysis workflows without writing code. At its core, it offers a widget-based visual programming environment where users connect configurable components to perform data preprocessing, machine learning model training, statistical evaluation, and interactive visualization. The platform is built on NumPy-backed data tables with domain descriptors that define variable names, types, and roles, and includes a lazy SQL query proxy for working with database tables without loading all data into memory. The

ljpzzz/machinelearning

8,706View on GitHub

This project is a machine learning implementation library featuring a collection of code examples that implement supervised, unsupervised, and reinforcement learning algorithms from scratch. It provides a comprehensive set of toolkits for core machine learning components, including a natural language processing toolkit, a reinforcement learning framework, and suites for data dimensionality reduction and pattern mining. The library includes specialized implementations for reinforcement learning, such as Q-Learning, Deep Q-Networks, and Actor-Critic agents. The natural language processing capab

Jack-Cherish/Machine-Learning

10,333View on GitHub

This project is a collection of supervised and unsupervised machine learning algorithms implemented from scratch using Python. It serves as an educational resource for studying model training, parameter optimization, and the implementation of core predictive models. The library provides a variety of supervised learning tools, including linear and logistic regression, decision trees, and support vector machines. It also features unsupervised learning capabilities for discovering patterns in unlabeled datasets through clustering algorithms. Broad capability areas include ensemble learning thro

WenDesilihang_book_algorithm

Features