1 repo
Awesome GitHub RepositoriesLong Context Training Optimizations
Techniques for training machine learning models on extended input sequences by managing memory and activation offloading.
Distinguishing note: Focuses specifically on memory-efficient handling of long-sequence data in transformer models.
Explore 1 awesome GitHub repository matching artificial intelligence & ml · Long Context Training Optimizations. Refine with filters or upvote what's useful.
Awesome Long Context Training Optimizations GitHub Repositories
Find the best repos with AI.We'll search the best matching repositories with AI.
- deepspeedai/DeepSpeed
deepspeedai/DeepSpeed
41,638View on GitHubDeepSpeed is a high-performance library designed to scale deep learning model training and inference across massive clusters of GPUs and compute nodes. It provides a comprehensive suite of tools for distributed training, enabling the execution of models that exceed the memory capacity of single devices through advanced parameter partitioning, pipeline-based model parallelism, and memory-efficient state offloading. The framework distinguishes itself through specialized communication-efficient optimizers and hardware-aware acceleration techniques. By utilizing gradient compression, quantization
The framework reduces memory usage while maintaining precision by chunking input sequences and offloading activations between GPU and host memory.
Pythonbillion-parameterscompressiondata-parallelism
