deepspeedai/DeepSpeedExamples
DeepSpeedExamples
DeepSpeedExamples is a collection of reference implementations and scripts for training, fine-tuning, and executing inference on large-scale AI models using DeepSpeed optimization. It provides a distributed model training guide and practical workflows for adapting large language models through memory-efficient techniques.
The repository includes specialized implementations for pipeline parallelism to handle models exceeding single GPU memory and a suite of examples for ZeRO memory optimization to reduce per-device overhead. It also features standardized test suites for benchmarking the throughput and latency of models running on DeepSpeed inference engines.
The project covers broad capability areas including GPU memory optimization, distributed AI benchmarking, and high-performance model inference. It demonstrates the use of weight compression and distributed optimization to scale neural networks across multiple computing nodes.
Features
- Distributed Memory Optimizers - Implements Zero Redundancy Optimizer (ZeRO) to partition model states and gradients across distributed GPUs.
- Reference Implementations - Provides functional application examples and codebases that serve as standardized models for implementing distributed AI training and inference.
- Distributed Training - Offers frameworks and utilities for scaling model training across multiple processors, GPUs, or nodes using distributed optimization.
- Data-Parallel Training - Implements data-parallel training strategies to synchronize gradients across multiple compute nodes.
- Distributed Training - Demonstrates how to configure data and model parallelism to train large neural networks across multiple nodes.
- GPU Memory Optimizers - Manages optimizer states and model weights across CPU and GPU memory to optimize VRAM usage.
- Inference Benchmarking Tools - Includes utilities for measuring processing speed, latency, and performance metrics of machine learning models across various hardware configurations.
- Large Language Model Fine-Tuning - Provides practical workflows for adapting pre-trained large language models to specific tasks using distributed optimization.
- Large-Scale Model Training - Provides scripts and implementations for training large-scale models that exceed single-device memory capacity.
- Inference Optimizations - Uses specialized inference engines to reduce latency and increase throughput for model predictions.
- Model Parallelism Frameworks - Provides reference implementations for dividing neural network layers across multiple devices using pipeline parallelism.
- Optimizer State Offloading - Implements mechanisms to move optimizer states from GPU memory to system RAM to support massive scales.
- Mixed Precision Training - Provides implementations for using 16-bit and 32-bit precision to reduce memory usage during training.
- Model Performance Benchmarking - Includes standardized test suites to measure and compare the execution speed and efficiency of model implementations.
- Model Inference - Includes workflows for executing predictions on trained models across diverse hardware configurations.
- Model Compression - Includes techniques for reducing the memory footprint and computational requirements of large models.
- Pipeline Parallelism Partitioners - Implements utilities for partitioning large neural networks into sequential layers across multiple GPUs to enable pipeline-parallel training.
- Weight Quantization - Demonstrates weight compression and quantization to improve inference deployment efficiency and speed.
- Model Execution Benchmarks - Provides tools for benchmarking the computational efficiency and hardware utilization of model execution.
- Natural Language Processing - Listed in the “Natural Language Processing” section of the FunNLP awesome list.
