Awesome GitHub RepositoriesDiffusion Conditioning Architectures

ControlNet is a framework for structural image generation that extends pre-trained diffusion models with neural network architectures designed for precise spatial control. By injecting structural guidance directly into the latent-space denoising process, the system enables users to enforce geometric or semantic constraints on generated outputs while maintaining style consistency. The framework distinguishes itself through a weight-locked copying mechanism that preserves the integrity of the original model while introducing new control signals. It supports multi-condition synthesis, allowing f

This is a framework for training and sampling diffusion models to generate high-fidelity images, video, and 4D assets. It provides a modular environment for managing generative AI training pipelines, including the handling of datasets, noise sampling, and loss weighting to stabilize the creation of synthetic content. The project features a modular model configuration system that uses YAML-based assembly to define network submodules and conditioners. It also includes a dedicated toolset for AI image watermarking, allowing for the embedding and detection of invisible markers to verify the origi

MiniCPM-o is a multimodal large language model designed to function as a real-time conversational assistant on edge devices. By mapping text, image, video, and audio inputs into a unified latent space, the system enables simultaneous cross-modal reasoning and full-duplex interaction. It is built as an edge-side inference engine, utilizing quantized model weights to maintain high-performance processing on consumer hardware. The system distinguishes itself through its integrated speech synthesis and voice cloning capabilities, which allow for the generation of expressive, personalized vocal out

Audiocraft is a deep learning audio library and machine learning framework designed for training, fine-tuning, and evaluating generative models for music and sound effects. It functions as a text-to-music generative model and a neural audio codec, providing the tools necessary to compress audio signals into discrete representations and synthesize high-fidelity waveforms from textual descriptions. The framework is distinguished by its ability to combine multiple conditioning signals, allowing for the generation of audio based on text prompts, melodic excerpts, or style-based audio clips. It al

Tortoise-tts is a neural text-to-speech engine and voice cloning toolkit designed for high-quality audio generation. It functions as a zero-shot synthesis system, meaning it can generate speech for unseen speakers without requiring additional training or fine-tuning for each new voice. The system specializes in replicating human vocal characteristics using small sets of reference audio clips. It allows for the extraction of voice latents to mimic specific speakers, the generation of random synthetic identities, and the blending of multiple voice profiles to create hybrid vocal identities. Th

AnimateAnyone is an appearance-preserving video synthesizer designed for character animation from a single static image. It functions as a diffusion image-to-video generator that transforms a source image into a high-fidelity video sequence while maintaining consistent character identity, clothing, and visual details across all frames. The system enables video-driven character reenactment by transferring motions, facial expressions, and body movements from a reference video onto a static character. It employs pose-guided video generation to control movement via skeleton keypoints and pose sig

CSM is a conversational speech generation model and text-to-speech engine that converts text and audio inputs into synthetic speech. It utilizes a large language model architecture to predict and decode audio tokens for voice synthesis. The system functions as a zero-shot voice cloner, replicating specific speaker identities using short audio samples without requiring additional training. This enables precise control over speaker identity and the creation of synthetic speech that mimics a specific person. The model covers conversational speech synthesis and text-to-speech generation, transfo

Stable Diffusion WebUI Forge is a web-based interface and inference engine designed for the generation of AI media. It functions as a platform for executing diffusion-based models, providing a centralized environment to manage image preprocessors, custom generation logic, and hardware-accelerated sampling. The project distinguishes itself through a neural network patching framework that allows for the modification of model layers and the application of spatial conditioning during inference. By injecting custom logic and adapters directly into the network, users can influence output behaviors

Neutts is a neural text-to-speech engine designed for real-time streaming output on edge devices such as phones and laptops. It supports voice cloning from short audio references, enabling zero-shot reproduction of a target speaker's voice, and can be fine-tuned or retrained from scratch for custom voices and styles. The system distinguishes itself through a decoder-only architecture that halves memory and accelerates generation on constrained hardware, combined with quantized model inference for reduced memory footprint. Its streaming decoder loop interleaves synthesis with playback, deliver

LatentSync es un generador de video impulsado por audio y modelo de difusión latente de sincronización labial diseñado para sincronizar los movimientos labiales de un hablante en un video con una pista de audio objetivo. Proporciona un framework de entrenamiento de sincronización labial para desarrollar redes de sincronización en datasets personalizados de video y audio. El sistema utiliza un pipeline de preprocesamiento de video para limpiar, segmentar y alinear datos faciales. Incluye una herramienta de evaluación de sincronización visual que calcula puntuaciones de confianza para medir la precisión de la alineación de audio y video en los videos generados. El proyecto cubre capacidades para el desarrollo de redes de sincronización personalizadas, gestión de configuración de entrenamiento para memoria de hardware y resolución, y evaluación de video sintético.

Parler-TTS is a library for generating high-quality speech from text, supporting both inference and model training. It combines a transformer-based text-to-speech generator with a mel-spectrogram decoder to convert written text into natural-sounding audio. The project distinguishes itself through text-conditioned voice control, which allows speaker attributes like gender, pitch, speaking rate, and style to be adjusted via a natural-language description. It also includes speaker embedding selection for maintaining voice identity across multiple generations, and a fine-tuning recipe system that

EchoMimic V2 es un pipeline de generación de video por IA y modelo de animación de visión por computadora diseñado para producir animaciones humanas sintéticas. Funciona como un framework generativo que crea videos de medio cuerpo alineando una imagen de referencia estática con movimientos de pose extraídos de un video de conducción. El sistema utiliza un proceso de generación basado en difusión combinado con compresión de espacio latente y un mecanismo de atención temporal para garantizar transiciones suaves entre fotogramas. Mantiene una identidad de persona consistente a través de codificación basada en referencia y guía la colocación espacial mediante condicionamiento de movimiento impulsado por pose. El proyecto incluye capacidades para el refinamiento de imágenes en múltiples etapas para mejorar el detalle y la nitidez facial. También proporciona herramientas para la preparación de datasets de animación, incluyendo la descarga y preprocesamiento de datos de video en formatos requeridos para el entrenamiento e inferencia del modelo.

Diffusion Policy is a robot learning framework that uses diffusion models to map visual observations to precise action trajectories. It functions as an imitation learning toolkit and visuomotor policy learner, providing a system to train neural networks that replicate human behavior by generating robotic movements based on image and sensor data. The framework employs a conditional denoising process to sample sequences of robotic movements, allowing it to handle multimodal action distributions where multiple valid trajectories may exist for a single state. It utilizes score-based action modeli

Champ is a generative vision system and controllable image-to-video generator designed for human image animation. It uses a diffusion-based video synthesizer and 3D parametric guidance to transform a single reference image into a consistent sequence of motion based on external driving data. The framework distinguishes itself through a human pose transfer system that employs 3D body parametric extraction and coordinate-space alignment. This allows the model to map motion from a driving video to a reference person by adjusting for body scales and camera perspectives using depth and semantic con

EchoMimic is an audio-driven portrait animation framework and latent diffusion video generator. It transforms static reference images into dynamic talking head videos by synchronizing facial movements with audio tracks and motion drivers. The system functions as a hybrid motion synthesis engine that combines audio inputs and pose data. It utilizes a facial landmark motion controller to edit positioning markers, enabling precise synchronization and video-to-video pose transfer. The pipeline covers image-to-video animation through latent diffusion and facial landmark conditioning. This allows

ComfyUI-LTXVideo is a generative framework and ComfyUI custom node extension for synthesizing high-fidelity video. It utilizes a latent diffusion and transformer-based system to create cinematic clips from text, image, and audio inputs, providing a modular interface for precise control over subject behavior and temporal consistency. The tool distinguishes itself with production-grade capabilities, including the generation of High Dynamic Range video in linear formats such as ARRI LogC3. It supports multimodal synchronization for audio-driven animation and lip-syncing, and allows for the creat