# pku-yuangroup/open-sora-plan **Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/pku-yuangroup-open-sora-plan).** 12,163 stars · 1,066 forks · Python · MIT ## Links - GitHub: https://github.com/PKU-YuanGroup/Open-Sora-Plan - awesome-repositories: https://awesome-repositories.com/repository/pku-yuangroup-open-sora-plan.md ## Description Open-Sora-Plan is a text-to-video framework and distributed video training system. It utilizes a diffusion transformer architecture and large language model components to transform written descriptions or image prompts into high-quality video sequences. The system features a distributed infrastructure designed for large-scale video training and inference. It employs sequence parallelism to split high-resolution or long-duration video samples across multiple GPUs and uses a sparse attention mechanism to increase processing speed. The project includes capabilities for both text-to-video and image-to-video generation. It manages data processing through latent space video compression and variable resolution management to preserve original aspect ratios during training. ## Tags ### Artificial Intelligence & ML - [Text-to-Video Generators](https://awesome-repositories.com/f/artificial-intelligence-ml/generative-ai-resources/diffusion-visual-models/generative-ai-pipelines/text-to-video-generators.md) — Synthesizes high-quality motion sequences from descriptive text prompts using transformer architectures. ([source](https://github.com/PKU-YuanGroup/Open-Sora-Plan/blob/main/docs/Report-v1.2.0.md)) - [Distributed Training](https://awesome-repositories.com/f/artificial-intelligence-ml/distributed-training.md) — Provides high-performance infrastructure for scaling video model training across multiple GPUs. - [Video Compression](https://awesome-repositories.com/f/artificial-intelligence-ml/generative-ai-resources/diffusion-visual-models/generative-ai-models/latent-space-generative-models/video-compression.md) — Compresses raw video frames into a lower-dimensional latent space to significantly reduce training costs. - [Large-Scale Model Training](https://awesome-repositories.com/f/artificial-intelligence-ml/large-scale-model-training.md) — Optimizes deep learning models for video generation that exceed single-device memory capacity. - [Sequence Parallelism Frameworks](https://awesome-repositories.com/f/artificial-intelligence-ml/sequence-parallelism-frameworks.md) — Uses sequence parallelism to distribute long video sequences across multiple GPUs to handle memory constraints. - [Sparse Attention Modules](https://awesome-repositories.com/f/artificial-intelligence-ml/sparse-attention-modules.md) — Implements sparse attention modules to reduce computational complexity and increase processing speed. - [Diffusion Transformers](https://awesome-repositories.com/f/artificial-intelligence-ml/transformer-architectures/diffusion-transformers.md) — Combines transformer-based attention with diffusion denoising to generate spatial-temporal video data. - [Video Generation](https://awesome-repositories.com/f/artificial-intelligence-ml/video-generation.md) — Implements a deep learning model that transforms text and image prompts into high-quality video. - [Image-to-Video Generation](https://awesome-repositories.com/f/artificial-intelligence-ml/video-generation/image-to-video-generation.md) — Synthesizes fluid video sequences using a reference image as a primary input. ([source](https://github.com/PKU-YuanGroup/Open-Sora-Plan/blob/main/docs/Report-v1.2.0.md)) - [Aspect Ratio Bucketing](https://awesome-repositories.com/f/artificial-intelligence-ml/aspect-ratio-bucketing.md) — Implements aspect ratio bucketing to maintain original frame proportions during training without forced cropping. - [Distributed Training Scaling Utilities](https://awesome-repositories.com/f/artificial-intelligence-ml/distributed-training-scaling-utilities.md) — Scales training and inference workloads for long-duration video samples across distributed GPU systems. ([source](https://github.com/PKU-YuanGroup/Open-Sora-Plan/blob/main/docs/Report-v1.2.0.md)) - [Generative Model Training Tools](https://awesome-repositories.com/f/artificial-intelligence-ml/generative-model-training-tools.md) — Ships tools to build and optimize generative models for high-quality video synthesis. ([source](https://github.com/pku-yuangroup/open-sora-plan#readme)) - [Variable Resolution Handling](https://awesome-repositories.com/f/artificial-intelligence-ml/inference-optimizations/aspect-ratio-optimizers/variable-resolution-handling.md) — Handles varying frame sizes and aspect ratios during the training process. ([source](https://github.com/PKU-YuanGroup/Open-Sora-Plan/blob/main/docs/Report-v1.2.0.md)) - [Multi-GPU Distribution](https://awesome-repositories.com/f/artificial-intelligence-ml/model-optimization/inference-deployment/model-deployment-toolkits/distributed-deployment-utilities/multi-gpu-distribution.md) — Splits high-resolution video samples across multiple GPUs to accelerate inference through sequence parallelism. - [Multilingual Text Embeddings](https://awesome-repositories.com/f/artificial-intelligence-ml/multilingual-text-embeddings.md) — Maps prompts from various languages into a shared vector space to guide the video generation process. ### Part of an Awesome List - [Training Data Compression](https://awesome-repositories.com/f/awesome-lists/media/video-processing/training-data-compression.md) — Reduces the size of video frames to lower training costs while preserving high image quality. ([source](https://github.com/pku-yuangroup/open-sora-plan#readme)) - [Foundation Models](https://awesome-repositories.com/f/awesome-lists/ai/foundation-models.md) — Open-source implementation of Sora-like video generation. - [Model Architectures](https://awesome-repositories.com/f/awesome-lists/ai/model-architectures.md) — Open-source reproduction of video generation architectures. - [Video Generation](https://awesome-repositories.com/f/awesome-lists/media/video-generation.md) — Community-driven project for open video generation. - [Video Generation Models](https://awesome-repositories.com/f/awesome-lists/media/video-generation-models.md) — Community-driven project for open video generation.