facebookresearch/dlrm
Dlrm
This is a PyTorch recommendation framework and deep learning recommendation model designed to generate personalized content predictions. It functions as a distributed embedding trainer that processes dense and sparse features through a neural network architecture to predict user preferences.
The project implements a CUDA-optimized machine learning system using specialized GPU kernels to accelerate embedding lookup and aggregation. It employs a distributed approach to shard massive sparse feature tables across multiple GPUs, enabling the training of large-scale models.
The system utilizes a two-tower architecture for feature interaction and supports hybrid parallelism, combining data and model parallelism across compute clusters. Its capability surface includes distributed training across network nodes, GPU memory optimization, and checkpoint-based state recovery.
Features
- Recommendation Model Training - Provides a framework for training recommendation models by processing dense and sparse features through deep learning architectures.
- Neural Recommendation Frameworks - Provides an end-to-end deep learning framework for building and training industrial-scale recommendation systems.
- Deep Learning Recommendation Systems - Implements a deep learning framework specifically for processing dense and sparse features to generate personalized predictions.
- Distributed ML Trainers - Implements a scalable system for distributing the training of large-scale embedding models across compute clusters.
- Distributed Training - Distributes model training across multiple GPUs and network nodes to process large-scale datasets.
- Embedding Table Sharding - Distributes large sparse feature tables across multiple GPU devices to fit memory and balance training loads.
- GPU Kernel Implementations - Provides a high-performance implementation using specialized GPU kernels to accelerate embedding lookup and aggregation.
- Embedding Lookup Kernels - Implements specialized GPU kernels to accelerate the lookup and aggregation of high-dimensional embedding vectors.
- GPU Memory Optimizers - Optimizes GPU VRAM usage using specialized kernels and sharding to manage high-dimensional embedding tables.
- Hybrid Model Parallelism - Combines data parallelism for dense layers and model parallelism for sparse embedding layers across compute clusters.
- Large-Scale Model Training - Distributes the training of complex architectures across multiple GPUs and network nodes to handle massive datasets.
- Synchronous Gradient Averaging - Uses synchronous communication primitives to ensure distributed workers maintain consistent model weights via gradient averaging.
- Preference Prediction - Analyzes user behavior and item characteristics to predict preferences and improve recommendation accuracy.
- Recommendation Architectures - Implements a neural network architecture designed to process dense and sparse features for personalized content prediction.
- Embedding Memory Layouts - Utilizes specialized GPU memory layouts to accelerate the retrieval and aggregation of high-dimensional embeddings.
- Dual-Tower Architectures - Processes dense and sparse features through separate networks before merging them via a dot product interaction layer.
- Training State Checkpointing - Implements mechanisms to save and restore the full training state, including optimizer parameters, to resume training from specific epochs.
- Machine Learning Workflow Libraries - Manages the distributed training process and model checkpoints across a cluster of machines for performance and stability.
- Model Checkpoints - Saves and loads model weights and optimizer states based on accuracy improvements to resume training.
- Recommender Systems - Deep learning model for large-scale recommendation and personalization.
