AberHu/Knowledge-Distillation-Zoo
Knowledge Distillation Zoo
Knowledge-Distillation-Zoo is a framework for neural network model compression that facilitates the transfer of learned patterns from large teacher models to smaller student architectures. It provides a modular environment for executing training pipelines designed to reduce the computational requirements of deep learning models while maintaining predictive accuracy.
The library implements knowledge transfer through both logit-based mimicry and feature-map alignment, allowing students to replicate the classification behavior and internal representations of a teacher. It supports teacher-student decoupling, where the teacher model remains frozen during the training process, and utilizes modular loss composition to balance task-specific objectives with distillation-specific penalties.
The toolkit includes a command-line interface for managing training workflows and supports configurable parameter injection to swap between different distillation strategies. It is built as a library for PyTorch, providing a structured environment for optimizing neural networks for deployment on resource-constrained hardware.
Features
- Neural Network Training Frameworks - Provides a modular environment for executing distillation pipelines that optimize student model performance.
- Model Distillation - Transfers learned patterns from a large teacher model into a smaller student model while maintaining accuracy.
- Knowledge Distillation - Applies loss functions and training strategies to teach compact models to mimic the performance of complex teacher models.
- Neural Network Model Compression - Reduces the size and computational requirements of deep learning models by transferring knowledge from large teacher networks.
- Deep Learning Toolkits - Provides a set of training strategies designed to reduce the computational requirements of neural networks while maintaining accuracy.
- Intermediate Representation Alignments - Minimizes the distance between intermediate layer activations to ensure the student learns internal representations similar to the teacher.
- Distillation Penalty Compositions - Combines standard task-specific objectives with distillation-specific penalty terms to balance accuracy and mimicry.
- Teacher-Student Distillation - Calculates divergence between teacher and student output probability distributions to force mimicry of classification behavior.
- Teacher-Student Decoupling Mechanisms - Maintains separate model architectures where the teacher remains frozen while the student updates its weights based on teacher guidance.
- Knowledge Distillation Pipelines - Includes a CLI-based training pipeline for student networks using pre-defined distillation methods and configurable hyperparameters.
- PyTorch Model Optimizations - Implements training pipelines to improve the efficiency of neural networks for deployment on resource-constrained hardware.
- Training Pipelines - Provides structured training pipelines that allow for configurable distillation strategies and loss functions via parameter injection.
Star history
How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
