PKU-YuanGroup/Open-Sora-Plan
Open Sora Plan
Open-Sora-Plan is a text-to-video framework and distributed video training system. It utilizes a diffusion transformer architecture and large language model components to transform written descriptions or image prompts into high-quality video sequences.
The system features a distributed infrastructure designed for large-scale video training and inference. It employs sequence parallelism to split high-resolution or long-duration video samples across multiple GPUs and uses a sparse attention mechanism to increase processing speed.
The project includes capabilities for both text-to-video and image-to-video generation. It manages data processing through latent space video compression and variable resolution management to preserve original aspect ratios during training.
Features
- Text-to-Video Generators - Synthesizes high-quality motion sequences from descriptive text prompts using transformer architectures.
- Distributed Training - Provides high-performance infrastructure for scaling video model training across multiple GPUs.
- Video Compression - Compresses raw video frames into a lower-dimensional latent space to significantly reduce training costs.
- Large-Scale Model Training - Optimizes deep learning models for video generation that exceed single-device memory capacity.
- Sequence Parallelism Frameworks - Uses sequence parallelism to distribute long video sequences across multiple GPUs to handle memory constraints.
- Sparse Attention Modules - Implements sparse attention modules to reduce computational complexity and increase processing speed.
- Diffusion Transformers - Combines transformer-based attention with diffusion denoising to generate spatial-temporal video data.
- Video Generation - Implements a deep learning model that transforms text and image prompts into high-quality video.
- Image-to-Video Generation - Synthesizes fluid video sequences using a reference image as a primary input.
- Aspect Ratio Bucketing - Implements aspect ratio bucketing to maintain original frame proportions during training without forced cropping.
- Distributed Training Scaling Utilities - Scales training and inference workloads for long-duration video samples across distributed GPU systems.
- Generative Model Training Tools - Ships tools to build and optimize generative models for high-quality video synthesis.
- Variable Resolution Handling - Handles varying frame sizes and aspect ratios during the training process.
- Multi-GPU Distribution - Splits high-resolution video samples across multiple GPUs to accelerate inference through sequence parallelism.
- Multilingual Text Embeddings - Maps prompts from various languages into a shared vector space to guide the video generation process.
- Training Data Compression - Reduces the size of video frames to lower training costs while preserving high image quality.
- Foundation Models - Open-source implementation of Sora-like video generation.
- Model Architectures - Open-source reproduction of video generation architectures.
- Video Generation - Community-driven project for open video generation.
- Video Generation Models - Community-driven project for open video generation.
