Awesome GitHub RepositoriesDiffusion Conditioning Architectures

ControlNet is a framework for structural image generation that extends pre-trained diffusion models with neural network architectures designed for precise spatial control. By injecting structural guidance directly into the latent-space denoising process, the system enables users to enforce geometric or semantic constraints on generated outputs while maintaining style consistency. The framework distinguishes itself through a weight-locked copying mechanism that preserves the integrity of the original model while introducing new control signals. It supports multi-condition synthesis, allowing f

This is a framework for training and sampling diffusion models to generate high-fidelity images, video, and 4D assets. It provides a modular environment for managing generative AI training pipelines, including the handling of datasets, noise sampling, and loss weighting to stabilize the creation of synthetic content. The project features a modular model configuration system that uses YAML-based assembly to define network submodules and conditioners. It also includes a dedicated toolset for AI image watermarking, allowing for the embedding and detection of invisible markers to verify the origi

MiniCPM-o is a multimodal large language model designed to function as a real-time conversational assistant on edge devices. By mapping text, image, video, and audio inputs into a unified latent space, the system enables simultaneous cross-modal reasoning and full-duplex interaction. It is built as an edge-side inference engine, utilizing quantized model weights to maintain high-performance processing on consumer hardware. The system distinguishes itself through its integrated speech synthesis and voice cloning capabilities, which allow for the generation of expressive, personalized vocal out

Audiocraft is a deep learning audio library and machine learning framework designed for training, fine-tuning, and evaluating generative models for music and sound effects. It functions as a text-to-music generative model and a neural audio codec, providing the tools necessary to compress audio signals into discrete representations and synthesize high-fidelity waveforms from textual descriptions. The framework is distinguished by its ability to combine multiple conditioning signals, allowing for the generation of audio based on text prompts, melodic excerpts, or style-based audio clips. It al

Tortoise-tts is a neural text-to-speech engine and voice cloning toolkit designed for high-quality audio generation. It functions as a zero-shot synthesis system, meaning it can generate speech for unseen speakers without requiring additional training or fine-tuning for each new voice. The system specializes in replicating human vocal characteristics using small sets of reference audio clips. It allows for the extraction of voice latents to mimic specific speakers, the generation of random synthetic identities, and the blending of multiple voice profiles to create hybrid vocal identities. Th

AnimateAnyone is an appearance-preserving video synthesizer designed for character animation from a single static image. It functions as a diffusion image-to-video generator that transforms a source image into a high-fidelity video sequence while maintaining consistent character identity, clothing, and visual details across all frames. The system enables video-driven character reenactment by transferring motions, facial expressions, and body movements from a reference video onto a static character. It employs pose-guided video generation to control movement via skeleton keypoints and pose sig

CSM is a conversational speech generation model and text-to-speech engine that converts text and audio inputs into synthetic speech. It utilizes a large language model architecture to predict and decode audio tokens for voice synthesis. The system functions as a zero-shot voice cloner, replicating specific speaker identities using short audio samples without requiring additional training. This enables precise control over speaker identity and the creation of synthetic speech that mimics a specific person. The model covers conversational speech synthesis and text-to-speech generation, transfo

Stable Diffusion WebUI Forge is a web-based interface and inference engine designed for the generation of AI media. It functions as a platform for executing diffusion-based models, providing a centralized environment to manage image preprocessors, custom generation logic, and hardware-accelerated sampling. The project distinguishes itself through a neural network patching framework that allows for the modification of model layers and the application of spatial conditioning during inference. By injecting custom logic and adapters directly into the network, users can influence output behaviors

Neutts is a neural text-to-speech engine designed for real-time streaming output on edge devices such as phones and laptops. It supports voice cloning from short audio references, enabling zero-shot reproduction of a target speaker's voice, and can be fine-tuned or retrained from scratch for custom voices and styles. The system distinguishes itself through a decoder-only architecture that halves memory and accelerates generation on constrained hardware, combined with quantized model inference for reduced memory footprint. Its streaming decoder loop interleaves synthesis with playback, deliver

LatentSync एक ऑडियो-ड्रिवन वीडियो जनरेटर और लेटेंट डिफ़्यूज़न लिप सिंक मॉडल है जिसे वीडियो में स्पीकर के होंठों की गतिविधियों को टारगेट ऑडियो ट्रैक के साथ सिंक्रोनाइज़ करने के लिए डिज़ाइन किया गया है। यह कस्टम वीडियो और ऑडियो डेटासेट पर सिंक्रोनाइज़ेशन नेटवर्क विकसित करने के लिए एक लिप सिंक्रोनाइज़ेशन ट्रेनिंग फ़्रेमवर्क प्रदान करता है। यह सिस्टम फेस डेटा को साफ़ करने, सेगमेंट करने और संरेखित करने के लिए एक वीडियो प्रीप्रोसेसिंग पाइपलाइन का उपयोग करता है। इसमें एक विज़ुअल सिंक मूल्यांकन टूल शामिल है जो जेनरेट किए गए वीडियो में ऑडियो और विज़ुअल संरेखण की सटीकता को मापने के लिए कॉन्फ़िडेंस स्कोर की गणना करता है। यह प्रोजेक्ट कस्टम सिंक्रोनाइज़ेशन नेटवर्क विकास, हार्डवेयर मेमोरी और रिज़ॉल्यूशन के लिए ट्रेनिंग कॉन्फ़िगरेशन मैनेजमेंट और सिंथेटिक वीडियो मूल्यांकन के लिए क्षमताओं को कवर करता है।

Parler-TTS is a library for generating high-quality speech from text, supporting both inference and model training. It combines a transformer-based text-to-speech generator with a mel-spectrogram decoder to convert written text into natural-sounding audio. The project distinguishes itself through text-conditioned voice control, which allows speaker attributes like gender, pitch, speaking rate, and style to be adjusted via a natural-language description. It also includes speaker embedding selection for maintaining voice identity across multiple generations, and a fine-tuning recipe system that

EchoMimic V2 एक AI वीडियो जनरेशन पाइपलाइन और कंप्यूटर विज़न एनिमेशन मॉडल है, जिसे सिंथेटिक ह्यूमन एनिमेशन बनाने के लिए डिज़ाइन किया गया है। यह एक जेनेरेटिव फ्रेमवर्क के रूप में काम करता है जो एक स्थिर रेफरेंस इमेज को ड्राइविंग वीडियो से निकाले गए पोज़ मूवमेंट के साथ अलाइन करके सेमी-बॉडी वीडियो तैयार करता है। यह सिस्टम फ्रेम के बीच स्मूथ ट्रांज़िशन सुनिश्चित करने के लिए डिफ्यूज़न-आधारित जनरेशन प्रक्रिया के साथ लेटेंट स्पेस कम्प्रेशन और टेम्पोरल अटेंशन मैकेनिज्म का उपयोग करता है। यह रेफरेंस-आधारित एन्कोडिंग के माध्यम से व्यक्ति की पहचान को बनाए रखता है और पोज़-ड्रिवन मोशन कंडीशनिंग के जरिए स्थानिक प्लेसमेंट को गाइड करता है। इस प्रोजेक्ट में चेहरे के विवरण और शार्पनेस को बेहतर बनाने के लिए मल्टी-स्टेज इमेज रिफाइनमेंट की क्षमताएं शामिल हैं। यह एनिमेशन डेटासेट तैयार करने के लिए भी टूल्स प्रदान करता है, जिसमें मॉडल ट्रेनिंग और इन्फरेंस के लिए आवश्यक फॉर्मेट में वीडियो डेटा को डाउनलोड और प्रीप्रोसेस करना शामिल है।

Diffusion Policy is a robot learning framework that uses diffusion models to map visual observations to precise action trajectories. It functions as an imitation learning toolkit and visuomotor policy learner, providing a system to train neural networks that replicate human behavior by generating robotic movements based on image and sensor data. The framework employs a conditional denoising process to sample sequences of robotic movements, allowing it to handle multimodal action distributions where multiple valid trajectories may exist for a single state. It utilizes score-based action modeli

Champ is a generative vision system and controllable image-to-video generator designed for human image animation. It uses a diffusion-based video synthesizer and 3D parametric guidance to transform a single reference image into a consistent sequence of motion based on external driving data. The framework distinguishes itself through a human pose transfer system that employs 3D body parametric extraction and coordinate-space alignment. This allows the model to map motion from a driving video to a reference person by adjusting for body scales and camera perspectives using depth and semantic con

EchoMimic is an audio-driven portrait animation framework and latent diffusion video generator. It transforms static reference images into dynamic talking head videos by synchronizing facial movements with audio tracks and motion drivers. The system functions as a hybrid motion synthesis engine that combines audio inputs and pose data. It utilizes a facial landmark motion controller to edit positioning markers, enabling precise synchronization and video-to-video pose transfer. The pipeline covers image-to-video animation through latent diffusion and facial landmark conditioning. This allows

ComfyUI-LTXVideo is a generative framework and ComfyUI custom node extension for synthesizing high-fidelity video. It utilizes a latent diffusion and transformer-based system to create cinematic clips from text, image, and audio inputs, providing a modular interface for precise control over subject behavior and temporal consistency. The tool distinguishes itself with production-grade capabilities, including the generation of High Dynamic Range video in linear formats such as ARRI LogC3. It supports multimodal synchronization for audio-driven animation and lip-syncing, and allows for the creat