Awesome GitHub RepositoriesDiffusion Transformers

Open-Sora is a video generation framework designed to produce cinematic sequences from text prompts and images. It functions as a generative system that transforms written descriptions or reference images into video content featuring realistic textures and lighting. The project includes a dedicated prompt engineering tool that uses large language models to expand simple user inputs into detailed descriptions. It also features a motion controller for adjusting movement intensity in generated sequences and evaluating motion levels in existing video files. The framework incorporates text-to-vid

F5-TTS is a text-to-speech system that utilizes a flow matching engine and diffusion transformers to generate fluent synthetic speech. It functions as a multilingual speech synthesizer and neural training framework, providing tools for voice cloning and high-performance inference serving. The project distinguishes itself through a voice cloning toolkit capable of mimicking specific speaker characteristics and tones from reference audio clips. It supports cross-lingual generation, allowing for the synthesis of audio across various global languages or the mixing of multiple languages within a s

Open-Sora-Plan is a text-to-video framework and distributed video training system. It utilizes a diffusion transformer architecture and large language model components to transform written descriptions or image prompts into high-quality video sequences. The system features a distributed infrastructure designed for large-scale video training and inference. It employs sequence parallelism to split high-resolution or long-duration video samples across multiple GPUs and uses a sparse attention mechanism to increase processing speed. The project includes capabilities for both text-to-video and im

DiT is a latent diffusion model and transformer-based generative AI framework implemented in PyTorch. It functions as a class-conditional image generator that replaces traditional convolutional backbones with a transformer architecture to synthesize high-fidelity images. The project utilizes patch-based latent processing and latent space compression to operate on low-dimensional image representations. It incorporates class-conditional guidance and adjustable guidance scales to control the visual content of generated images during the sampling process. The framework covers distributed model t

Sana is a framework for high-resolution image and video synthesis based on a linear diffusion transformer. It provides a toolkit for the training, fine-tuning, and execution of text-to-image and text-to-video models, as well as a video generative world model capable of simulating physical environments with precise spatial control. The project is distinguished by its use of linear complexity layers to handle high resolutions and its support for long-form, minute-length video generation in real time. It implements a two-stage inference paradigm that separates structural generation from visual t

StyleTTS2 is an adversarial text-to-speech model that uses style diffusion and large speech language models to generate natural-sounding speech from text input. It combines adversarial training with large pre-trained speech models to improve speech quality and reduce artifacts, while employing a style diffusion process that extracts prosodic and timbral features from reference audio to guide speech generation. The model supports multi-speaker voice synthesis by conditioning the diffusion process on speaker-specific embeddings derived from reference utterances, enabling voice cloning and adapt

YuE: Open Full-song Music Generation Foundation Model, something similar to Suno.ai but open

MegaTTS3 is a bilingual speech synthesis system that generates natural-sounding speech in Chinese and English, including seamless code-switching within a single utterance. It functions as a text-to-speech engine, voice cloning system, and speech-to-text alignment tool, built around an acoustic latent compression model that encodes high-resolution audio into compact representations for efficient processing. The system distinguishes itself through accent intensity control, allowing adjustment of a speaker's accent strength in generated speech, and voice cloning from short audio samples for pers

HunyuanDiT هو نموذج توليدي ثنائي اللغة لتحويل النص إلى صورة ومولد صور يعتمد على محولات الانتشار (diffusion transformer). يستخدم نظام انتشار كامن لتوليف صور عالية الدقة من مطالبات نصية، مع تركيز خاص على فهم وتوليد المحتوى من الأوصاف باللغتين الصينية والإنجليزية. يتميز المشروع ببنية محول متعددة الدقة ومساحة تضمين ثنائية اللغة لربط نصوص مختلفة في منطقة دلالية مشتركة. يدعم تحسين الصور التكراري متعدد الجولات، والذي يترجم الحوار التفاعلي إلى مطالبات محدثة لتعديل المحتوى المرئي تدريجياً. يتضمن النظام قدرات للتعليق التوضيحي التلقائي للصور، وقيود هيكلية للصور للتحكم في التخطيط، وضبط أوزان النموذج لتكييف المولد مع مجموعات بيانات أو أنماط فنية محددة. تشمل تحسينات الأداء تقطير النموذج لتسريع الاستدلال ودعم التنفيذ على الأجهزة ذات ذاكرة الفيديو المنخفضة.

This is a PyTorch deep learning framework and tool for human motion synthesis that generates 3D character animations from text prompts or action descriptions. It functions as a text-to-motion generator that converts natural language and categorical labels into temporally consistent 3D skeletal movement sequences. The system utilizes a transformer-based diffusion model to iteratively denoise motion data. It includes capabilities for action-conditioned generation, monocular-to-3D motion lifting, and motion sequence editing using text constraints. The framework incorporates geometric motion con

ACE-Step is a high-fidelity audio synthesis system and diffusion model designed to generate music and vocals from text descriptions. It functions as a music generator and vocal synthesizer, using a diffusion transformer decoder to produce audio across various languages and genres. The project provides tools for text-guided audio editing, including the ability to extend the duration of tracks, regenerate specific song segments, and perform latent-space audio inpainting to modify lyrics or styles. It also includes a framework for audio style fine-tuning using low-rank adaptation to adapt vocal

هذا إطار عمل للتعلم الذاتي الإشراف في PyTorch مصمم لتدريب النماذج التي تتعلم تمثيلات بصرية من الفيديو. ينفذ معمارية تنبؤية ذات تضمين مشترك تستخرج ميزات مكانية وزمانية من خلال التنبؤ بالمناطق المفقودة من إشارة داخل مساحة تمثيل كامنة بدلاً من إعادة بناء البكسلات الخام. يتضمن المشروع أداة تصور للمساحة الكامنة تستخدم نموذج انتشار مشروط لفك تشفير تنبؤات مساحة الميزات مرة أخرى إلى بكسلات. وهذا يسمح بالتحقق من التمثيلات المتعلمة عن طريق تحويل التنبؤات المجردة إلى صور قابلة للتفسير. يوفر إطار العمل حزمة تدريب موزعة لتنفيذ التدريب المسبق والتقييم واسع النطاق عبر مجموعات GPU متعددة. ويغطي خط الإنتاج الكامل لتعلم تمثيل الفيديو، بما في ذلك أخذ عينات البيانات المكانية والزمانية، واستخراج الميزات القائم على المحولات، وتقييم المشفرات المجمدة من خلال الاختبار الخطي ومعايير التصنيف.

Mochi is an open-source text-to-video diffusion model designed to synthesize high-fidelity video sequences from natural language prompts. It utilizes a diffusion transformer architecture to generate temporal video data. The project includes a framework for low-rank adaptation, allowing the model to be fine-tuned on custom datasets to specialize visual styles or specific subjects. It also features a distributed inference engine that spreads model workloads across multiple graphics cards to increase memory capacity and processing speed. The system covers programmable video generation through a