Awesome GitHub RepositoriesTraining Memory Optimizers

Swin-Transformer is a deep learning framework designed for training and deploying hierarchical vision transformer models. It serves as a research library and toolkit for computer vision tasks, providing the infrastructure to build models that replace standard convolution operations with sliding window self-attention mechanisms. By utilizing a multi-scale feature hierarchy, the framework enables the processing of visual data at varying resolutions and spatial scales. The project distinguishes itself through its implementation of shifted window partitioning, which facilitates global information

Swift is a toolkit for the full-parameter and parameter-efficient fine-tuning of large language and multimodal models. It functions as a multimodal model trainer for text, image, video, and audio data, and includes specialized tools for model compression and reinforcement learning from human feedback. The framework provides an alignment toolkit for optimizing model behavior using preference learning algorithms and reinforcement learning. It integrates parameter-efficient fine-tuning methods to adapt models with minimal memory and compute requirements, alongside utilities for reducing hardware

MNN is a high-performance inference engine and framework designed for on-device machine learning. It provides a comprehensive environment for executing, optimizing, and deploying neural network models directly on mobile and resource-constrained edge devices. The framework distinguishes itself through a robust model optimization toolkit that supports quantization, compression, and structural graph manipulation to minimize memory footprint and maximize execution speed. It features a modular architecture that abstracts hardware-specific backends, allowing models to run efficiently across diverse

Axolotl is a distributed training orchestrator and fine-tuning framework for large language models, multimodal systems, and quantized models. It provides a structured environment for specializing pre-trained models through full parameter updates or low-rank adaptation, as well as aligning model outputs with human expectations via preference tuning pipelines and reward modeling. The system distinguishes itself through a configuration-driven pipeline that manages preprocessing and training workflows via a single file for reproducibility. It implements high-throughput optimizations such as multi

This project is a quantized fine-tuning framework for large language models. It implements a low-rank adaptation library and a four-bit quantizer to reduce the GPU memory requirements needed to train large models. The framework utilizes four-bit quantization and low-rank adapters to enable model training on consumer-grade hardware. It further reduces the memory footprint through double quantization and a paged optimizer that offloads states to system RAM. The system supports distributed training across multiple GPUs to handle larger parameter scales and includes utilities for custom dataset

OpenRLHF is a training framework and alignment library designed for reinforcement learning from human feedback across distributed GPU clusters. It provides tools for aligning large language models and multimodal vision-language models using algorithms such as PPO, GRPO, and DPO. The framework distinguishes itself through a distributed inference engine that overlaps sample rollout with training to increase throughput. It supports scaling to models exceeding 70 billion parameters via parameter sharding and handles long-context sequences through ring-attention sequence parallelism. The project

Sana is a framework for high-resolution image and video synthesis based on a linear diffusion transformer. It provides a toolkit for the training, fine-tuning, and execution of text-to-image and text-to-video models, as well as a video generative world model capable of simulating physical environments with precise spatial control. The project is distinguished by its use of linear complexity layers to handle high resolutions and its support for long-form, minute-length video generation in real time. It implements a two-stage inference paradigm that separates structural generation from visual t

bitsandbytes is a quantization library for large language models that reduces memory footprints using k-bit quantization. It provides a framework for 4-bit low-rank adaptation, tools for 8-bit model compression, and memory-efficient optimizer extensions for PyTorch. The project enables the training of large models on limited hardware through 4-bit quantization and low-rank adaptation weights. It also facilitates faster inference by compressing models to 8-bit precision using vector-wise quantization. The library covers a range of memory optimization capabilities, including optimizer memory r

bitsandbytes is a deep learning quantization tool and library designed to reduce the memory footprint of large language models. It serves as a GPU memory optimizer and quantization framework, compressing model weights and features to 8-bit and 4-bit precision to enable inference and training on hardware with limited memory. The project provides a framework for low-rank adaptation, allowing the fine-tuning of quantized models by combining 4-bit weights with small trainable matrices. It further distinguishes itself through memory paging, which moves optimizer states between CPU and GPU memory t

该项目是一个实现 Faster R-CNN 架构的 PyTorch 目标检测框架。它作为一种视觉模型，用于预测图像和实时视频流中多个目标周围的精确边界框。 该系统针对多 GPU 训练进行了优化，以减少模型收敛所需的时间。它利用 GPU 加速设计来处理复杂检测网络的训练和推理。 该框架涵盖了完整的目标检测生命周期，包括针对静态图像和实时视频流的自定义网络训练和推理。它包括使用标准化数据集进行模型性能验证的功能，以及诸如基于长宽比的分组和分布式工作负载等训练优化。

This project is a comprehensive educational curriculum and structured learning path covering the full lifecycle of large language models. It provides a guided progression through the theory, architecture, training, and deployment of these models. The curriculum includes specialized guides on transformer architecture, model training tutorials, and frameworks for designing autonomous agents. It also provides dedicated resources for studying model safety and ethics. The material covers a wide range of technical capabilities, including distributed training strategies, parameter-efficient fine-tu

DeepSpeedExamples is a collection of reference implementations and scripts for training, fine-tuning, and executing inference on large-scale AI models using DeepSpeed optimization. It provides a distributed model training guide and practical workflows for adapting large language models through memory-efficient techniques. The repository includes specialized implementations for pipeline parallelism to handle models exceeding single GPU memory and a suite of examples for ZeRO memory optimization to reduce per-device overhead. It also features standardized test suites for benchmarking the throug

Tensorpack 是一个高级 TensorFlow 神经网络框架和研究库，专为构建和训练深度学习模型而设计。它提供了一系列可复现的神经网络架构，用于计算机视觉、生成任务、强化学习和自然语言处理。 该项目通过一个专门的深度学习数据流水线脱颖而出，该流水线使用纯 Python 进行并行数据加载和流式传输。它包括一个用于通过数据并行策略分发工作负载的多 GPU 训练编排器，以及一个用于可视化模型显著性和激活图的专用可解释性工具包。 该框架涵盖了广泛的功能，包括用于目标检测和语义分割的计算机视觉流水线、用于语音和文本的序列建模，以及强化学习代理开发。它还提供用于权重量化和低位宽训练的模型优化工具，以及用于复现学术研究论文和转换遗留 Caffe 模型权重的实用程序。

Tensorpack 是一个高性能 TensorFlow 训练框架和分布式深度学习工具包。它提供了一套用于构建和训练神经网络的工具，重点在于执行速度和架构灵活性。 该项目作为一个神经网络优化套件，实现了高效的执行模式以减少训练开销。它作为一个并行数据加载流水线，利用自动化并行化在处理大数据集时最大化吞吐量。 该工具包涵盖了使用数据并行策略在多个 GPU 和计算集群上进行的分布式训练。其功能包括大规模数据集处理和性能优化，以提高训练吞吐量。

Liger-Kernel is a collection of pre-built fused Triton kernels and patching utilities designed to accelerate large language model training. It provides drop-in kernel replacements for common LLM operations such as RMSNorm, cross-entropy loss, and attention, enabling increased throughput and reduced memory usage while preserving bitwise-exact gradients. The project serves as a toolkit for composing custom model architectures from individual optimized kernels and for patching pre-existing models with minimal code changes. The project distinguishes itself through its ability to perform runtime m

Torchtune is a PyTorch-native library for fine-tuning, aligning, and quantizing large language models. It provides a config-driven system for instantiating components, orchestrating distributed training, and managing parameter-efficient fine-tuning with quantization support, all through YAML-based configurations and command-line overrides. The library distinguishes itself through its comprehensive post-training workflow orchestration, combining supervised fine-tuning, preference optimization (DPO, PPO, GRPO), knowledge distillation, and quantization-aware training in a single configurable pip

Torchtune is a PyTorch-native library for fine-tuning, aligning, and quantizing large language models. It provides a configurable training pipeline orchestrated through YAML recipes, with CLI overrides and component swapping, distributed training via FSDP2, memory optimizations, and parameter-efficient fine-tuning methods like LoRA, DoRA, and QLoRA. The library distinguishes itself through its YAML-driven configuration system that defines all training parameters and instantiates components from config files, with full CLI override capability for any field or component at launch time. It suppo

RF-DETR is a Python library for training and deploying object detection, instance segmentation, and keypoint detection models built on a vision transformer architecture. It provides a unified command-line interface and Python API for the full workflow, from fine-tuning pretrained checkpoints on custom datasets to running inference on images, video files, and live camera streams. The project supports training on datasets in COCO or YOLO format, with automatic format detection and configurable augmentation pipelines. Models can be exported to ONNX, TFLite, or TensorRT for deployment across edge

Composer 是一个 PyTorch 分布式训练框架，旨在实现大规模模型在多节点 GPU 集群上的扩展。它兼具大语言模型训练器、分布式模型优化器和训练生命周期管理器的功能。 该项目作为深度学习正则化库脱颖而出，提供诸如 Sharpness Aware Minimization、MixUp 和 CutMix 等专业优化技术，以提升模型的泛化能力。它还通过序列长度预热、渐进式层冻结以及用于大规模模型恢复的分片状态检查点技术，优化了训练流程。 该框架涵盖了广泛的功能领域，包括分布式训练编排、混合精度硬件管理和云原生数据流。它还为 GPU 内存诊断、训练发散检测和吞吐量跟踪提供了丰富的监控与可观测性工具。 该项目包含一个命令行启动器，可自动执行跨节点的分布式多 GPU 训练任务。