Awesome GitHub RepositoriesDiffusion Transformers

Open-Sora is a video generation framework designed to produce cinematic sequences from text prompts and images. It functions as a generative system that transforms written descriptions or reference images into video content featuring realistic textures and lighting. The project includes a dedicated prompt engineering tool that uses large language models to expand simple user inputs into detailed descriptions. It also features a motion controller for adjusting movement intensity in generated sequences and evaluating motion levels in existing video files. The framework incorporates text-to-vid

F5-TTS is a text-to-speech system that utilizes a flow matching engine and diffusion transformers to generate fluent synthetic speech. It functions as a multilingual speech synthesizer and neural training framework, providing tools for voice cloning and high-performance inference serving. The project distinguishes itself through a voice cloning toolkit capable of mimicking specific speaker characteristics and tones from reference audio clips. It supports cross-lingual generation, allowing for the synthesis of audio across various global languages or the mixing of multiple languages within a s

Open-Sora-Plan is a text-to-video framework and distributed video training system. It utilizes a diffusion transformer architecture and large language model components to transform written descriptions or image prompts into high-quality video sequences. The system features a distributed infrastructure designed for large-scale video training and inference. It employs sequence parallelism to split high-resolution or long-duration video samples across multiple GPUs and uses a sparse attention mechanism to increase processing speed. The project includes capabilities for both text-to-video and im

DiT 是一个在 PyTorch 中实现的潜在扩散模型和基于 Transformer 的生成式 AI 框架。它作为一个类条件图像生成器，用 Transformer 架构取代了传统的卷积骨干网络，以合成高保真图像。 该项目利用基于补丁的潜在处理和潜在空间压缩来对低维图像表示进行操作。它结合了类条件引导和可调引导尺度，以在采样过程中控制生成图像的视觉内容。 该框架涵盖分布式模型训练、迭代噪声采样和合成图像数据集的创建。它还包括用于模型质量评估的工具，以根据标准基准计算准确度和质量分数。

Sana is a framework for high-resolution image and video synthesis based on a linear diffusion transformer. It provides a toolkit for the training, fine-tuning, and execution of text-to-image and text-to-video models, as well as a video generative world model capable of simulating physical environments with precise spatial control. The project is distinguished by its use of linear complexity layers to handle high resolutions and its support for long-form, minute-length video generation in real time. It implements a two-stage inference paradigm that separates structural generation from visual t

StyleTTS2 is an adversarial text-to-speech model that uses style diffusion and large speech language models to generate natural-sounding speech from text input. It combines adversarial training with large pre-trained speech models to improve speech quality and reduce artifacts, while employing a style diffusion process that extracts prosodic and timbral features from reference audio to guide speech generation. The model supports multi-speaker voice synthesis by conditioning the diffusion process on speaker-specific embeddings derived from reference utterances, enabling voice cloning and adapt

YuE: Open Full-song Music Generation Foundation Model, something similar to Suno.ai but open

MegaTTS3 is a bilingual speech synthesis system that generates natural-sounding speech in Chinese and English, including seamless code-switching within a single utterance. It functions as a text-to-speech engine, voice cloning system, and speech-to-text alignment tool, built around an acoustic latent compression model that encodes high-resolution audio into compact representations for efficient processing. The system distinguishes itself through accent intensity control, allowing adjustment of a speaker's accent strength in generated speech, and voice cloning from short audio samples for pers

HunyuanDiT 是一个双语文本到图像生成模型和扩散 Transformer 图像生成器。它使用潜在扩散系统从文本提示中合成高分辨率图像，特别侧重于理解和生成来自中文和英文描述的内容。 该项目具有多分辨率 Transformer 架构和双语嵌入空间，可将不同的脚本映射到共享的语义区域。它支持迭代式多轮图像细化，将对话转换为更新后的提示，以逐步修改视觉内容。 该系统包括自动图像标注、用于布局控制的结构化图像约束，以及模型权重微调功能，以使生成器适应特定的数据集或艺术风格。性能优化包括用于推理加速的模型蒸馏，以及对低显存硬件上执行的支持。

这是一个 PyTorch 深度学习框架和人体运动合成工具，可从文本提示或动作描述生成 3D 角色动画。它作为一个文本到运动生成器，将自然语言和分类标签转换为时间上一致的 3D 骨骼运动序列。 该系统利用基于 Transformer 的扩散模型来迭代去噪运动数据。它包括动作条件生成、单目到 3D 运动提升以及使用文本约束进行运动序列编辑的功能。 该框架结合了几何运动约束强制执行，以通过关节位置和速度损失确保物理合理性。它进一步涵盖了完整的动画流水线，包括运动模型训练、针对基准数据集的性能评估、3D 网格渲染以及用于环境交互的基于物理的模拟控制。

ACE-Step is a high-fidelity audio synthesis system and diffusion model designed to generate music and vocals from text descriptions. It functions as a music generator and vocal synthesizer, using a diffusion transformer decoder to produce audio across various languages and genres. The project provides tools for text-guided audio editing, including the ability to extend the duration of tracks, regenerate specific song segments, and perform latent-space audio inpainting to modify lyrics or styles. It also includes a framework for audio style fine-tuning using low-rank adaptation to adapt vocal

这是一个 PyTorch 自监督学习框架，旨在训练从视频中学习视觉表示的模型。它实现了一种联合嵌入预测架构，通过预测潜在表示空间内信号的缺失区域而不是重建原始像素来提取时空特征。 该项目包括一个潜在空间可视化工具，使用条件扩散模型将特征空间预测解码回像素。这允许通过将抽象预测转换为可解释的图像来验证学习到的表示。 该框架提供了一个分布式训练套件，用于在多 GPU 集群上执行大规模预训练和评估。它涵盖了视频表示学习的端到端流水线，包括时空数据采样、基于 Transformer 的特征提取，以及通过线性探测和分类基准对冻结编码器进行评估。

Mochi is an open-source text-to-video diffusion model designed to synthesize high-fidelity video sequences from natural language prompts. It utilizes a diffusion transformer architecture to generate temporal video data. The project includes a framework for low-rank adaptation, allowing the model to be fine-tuned on custom datasets to specialize visual styles or specific subjects. It also features a distributed inference engine that spreads model workloads across multiple graphics cards to increase memory capacity and processing speed. The system covers programmable video generation through a