Awesome GitHub RepositoriesModel Distillation Methods

This project is a comprehensive framework for the entire lifecycle of transformer-based language models, supporting everything from foundational pretraining to specialized deployment. It provides a modular toolkit for defining neural network architectures, managing data preparation pipelines, and executing training routines across various scales. The framework is designed to handle the full model development process, including supervised fine-tuning, behavioral alignment, and the integration of agentic capabilities. What distinguishes this framework is its focus on efficient training and adva

This library provides a comprehensive collection of modular building blocks and research-backed architectures for implementing vision transformers within the PyTorch framework. It serves as a centralized repository for constructing, training, and analyzing attention-based models, offering a wide array of specialized variants designed for image classification and visual representation learning. The project distinguishes itself through a focus on architectural efficiency and flexibility, supporting diverse input formats including non-square images and volumetric data like video. It incorporates

This project is a distributed training infrastructure designed for aligning large language models through reinforcement learning. It functions as an end-to-end engine for complex alignment tasks, including proximal policy optimization, direct preference optimization, and iterative self-play. By providing a unified framework for multi-turn interactions and tool-use scenarios, it enables the development of models capable of reasoning and external environment engagement. The framework distinguishes itself through a decoupled architecture that separates model training from sample generation. This

MNN is a high-performance inference engine and framework designed for on-device machine learning. It provides a comprehensive environment for executing, optimizing, and deploying neural network models directly on mobile and resource-constrained edge devices. The framework distinguishes itself through a robust model optimization toolkit that supports quantization, compression, and structural graph manipulation to minimize memory footprint and maximize execution speed. It features a modular architecture that abstracts hardware-specific backends, allowing models to run efficiently across diverse

Open CLIP is an open source framework for training and deploying Contrastive Language-Image Pre-training models. It serves as a vision-language training framework and multimodal embedding engine that maps images and text into a shared vector space for similarity searches and zero-shot classification. The project provides a toolkit for distributed training of contrastive models and includes an image-to-text generative model for producing natural language descriptions. It supports custom text encoder integration and utilizes teacher-student model distillation to transfer knowledge from large pr

DINOv2 is a self-supervised vision transformer foundation model designed to generate high-quality visual representations from raw image data. By leveraging large-scale unlabelled datasets, the framework learns to extract robust numerical embeddings that serve as inputs for various machine learning and analysis workflows. The model distinguishes itself through a teacher-student training framework that utilizes centered and sharpened soft probability distributions to align feature maps across multiple image crops. It incorporates a masking strategy that forces the model to reconstruct missing i

ParlAI is a conversational AI research framework designed for training, evaluating, and sharing dialogue models using a unified interface for datasets and agents. It functions as a PyTorch-based training platform and a dialogue data collection system, providing a centralized model zoo for the distribution of versioned pretrained agents. The project distinguishes itself through a knowledge-grounded retrieval system that combines dense and sparse indexing to ground responses in external information. It also provides a comprehensive infrastructure for gathering human-AI interaction data via inte

This project is a self-supervised vision foundation model based on a vision transformer architecture. It is designed to learn dense visual representations from unlabeled images, serving as a general-purpose backbone for a wide variety of downstream vision tasks. The system is distinguished by its use of self-distillation and masked image modeling to extract semantic and geometric features. It also incorporates an image-text alignment model that maps visual embeddings to textual descriptions, enabling zero-shot image recognition, zero-shot segmentation, and cross-modal retrieval. The project

CleanRL is a reinforcement learning library and PyTorch framework providing a suite of reproducible implementations for online reinforcement learning algorithms. It serves as a deep reinforcement learning benchmark suite and experiment orchestrator designed for research and agent development across both discrete and continuous action spaces. The project is distinguished by its single-file algorithm implementation approach, which encapsulates each algorithm in a standalone script to eliminate complex class hierarchies. This structure is paired with a system for scheduling and executing large-s

Oumi is a comprehensive large language model development platform designed for synthesizing data, fine-tuning models, and running performance evaluations. It serves as a unified environment for the entire model lifecycle, encompassing a training and fine-tuning suite, an evaluation framework, and tools for synthetic data generation and model distillation. The platform is distinguished by its iterative, failure-driven synthesis approach, which analyzes model weaknesses during evaluation to generate targeted training data. It utilizes an LLM-based judge framework to programmatically score respo

Depth-Anything is a monocular depth estimation foundation model that produces dense per-pixel depth maps from a single RGB image. It is built on a DINOv2 Vision Transformer encoder backbone and trained on 62 million unlabeled images using a teacher-student pseudo-labeling framework, enabling robust generalization across diverse scenes without task-specific training. The model outputs both relative depth maps, which capture the ordering of scene points, and metric depth maps with real-world units after fine-tuning on datasets like NYUv2 or KITTI. The project distinguishes itself through its ab

This project is a PyTorch vision transformer framework designed for self-supervised learning. It implements a model that trains visual representations using a momentum teacher and self-distillation without the need for labeled data. The library functions as an image feature extractor and visual attention visualizer, allowing for the generation of high-dimensional vectors and the rendering of self-attention maps as heatmaps or videos to analyze model focus. It provides comprehensive tools for downstream vision evaluation, including linear probe classification, k-nearest neighbor categorizatio

This project is a toolkit for fine-tuning and managing text-to-image diffusion models. It focuses on low-rank adaptation to create small, portable weight files that customize model styles and behaviors without modifying the entire base model. The project provides specialized utilities for model distillation using singular value decomposition to extract adapters from fully trained models, as well as tools for blending and merging multiple adapters through weight interpolation. It includes capabilities for subject inversion and pivotal tuning to increase the visual fidelity of specific identiti

This project is a framework for training and sampling generative models designed to produce high-quality images in few steps. It provides implementations for image generation models that transform random noise into structured visual data through an optimized sampling process. The system specializes in accelerating image generation through consistency distillation and consistency training. It includes tools to transform pre-trained diffusion models into faster versions by distilling knowledge from a teacher model into a student model, as well as methods to train consistency models from scratch

Isaac Lab is an open-source framework for training robot policies in physically simulated environments, supporting both single-agent and multi-agent reinforcement learning. It is built on an Omniverse-PhysX simulation backend that models rigid bodies, articulated systems, deformable objects, and sensors, and provides a task-based environment configuration system where each training environment is defined as a modular class specifying observation spaces, action spaces, reward functions, and termination conditions. The framework distinguishes itself through an RL-library abstraction layer that

Torchtune is a PyTorch-native library for fine-tuning, aligning, and quantizing large language models. It provides a config-driven system for instantiating components, orchestrating distributed training, and managing parameter-efficient fine-tuning with quantization support, all through YAML-based configurations and command-line overrides. The library distinguishes itself through its comprehensive post-training workflow orchestration, combining supervised fine-tuning, preference optimization (DPO, PPO, GRPO), knowledge distillation, and quantization-aware training in a single configurable pip

Torchtune is a PyTorch-native library for fine-tuning, aligning, and quantizing large language models. It provides a configurable training pipeline orchestrated through YAML recipes, with CLI overrides and component swapping, distributed training via FSDP2, memory optimizations, and parameter-efficient fine-tuning methods like LoRA, DoRA, and QLoRA. The library distinguishes itself through its YAML-driven configuration system that defines all training parameters and instantiates components from config files, with full CLI override capability for any field or component at launch time. It suppo

OpenClaw-RL es un framework de aprendizaje por refuerzo para entrenar agentes de modelos de lenguaje grandes. Proporciona un sistema para optimizar las políticas de los agentes utilizando instrucciones en lenguaje natural, optimización basada en recompensas y un pipeline para refinar el comportamiento del modelo a través de retroalimentación humana o de jueces. El framework utiliza adaptación de bajo rango (LoRA) para actualizar los pesos del modelo de manera eficiente sin un ajuste completo de parámetros. Cuenta con una infraestructura local-first que permite que el modelo de política, el juez y el entrenador se ejecuten en hardware privado para mantener la localidad de los datos. El sistema cubre una gama de capacidades de optimización, incluyendo el entrenamiento de uso de herramientas para entornos de terminal y GUI y el refinamiento de políticas conversacionales. Implementa un bucle de entrenamiento híbrido que combina recompensas escalares con destilación a nivel de token para proporcionar retroalimentación tanto densa como dispersa. Para mantener el rendimiento, emplea una arquitectura de ejecución asíncrona que desacopla el servicio del agente del proceso de entrenamiento.

Super-Gradients es un framework de visión artificial de PyTorch y biblioteca de entrenamiento diseñada para el ciclo de vida completo de los modelos de visión. Funciona como un optimizador de modelos de deep learning y un kit de herramientas de despliegue para entrenar y ajustar modelos en tareas de clasificación de imágenes, detección de objetos, segmentación semántica y estimación de pose. El proyecto proporciona herramientas específicas para la optimización de modelos, incluyendo destilación de conocimiento profesor-estudiante y compresión de precisión numérica para reducir los requisitos de memoria y computación. También incluye la implementación de la arquitectura Yolo-NAS para detección de objetos de alto rendimiento. El framework cubre una amplia superficie de capacidades, incluyendo entrenamiento distribuido en GPU, pipelines de visión modulares y la automatización de ejecuciones de entrenamiento mediante configuraciones de recetas estructuradas. Además, gestiona la carga de datos, la aumentación de imágenes y la exportación de pesos entrenados a formatos universales para aceleradores de hardware de producción.

Co-tracker is a PyTorch point tracking framework and dense point tracking model designed to map the motion of individual pixels throughout a video. It functions as a video pixel tracker that predicts point trajectories and visibility masks across sequences of video frames. The project includes a computer vision training pipeline that utilizes teacher-student knowledge distillation. This allows for the generation of pseudo-labels from unannotated real video data to fine-tune pre-trained models and reduce the gap between synthetic and real data environments. The framework provides capabilities