CompVis/latent-diffusion
Latent Diffusion
Latent Diffusion is a framework for high-resolution image synthesis that performs the denoising process within a compressed latent space. It uses variational autoencoders to encode images into a lower-dimensional representation, reducing the computational cost of noise prediction compared to operating on raw pixels.
The project enables text-to-image generation by integrating natural language descriptions through cross-attention conditioning. It also supports image inpainting and restoration, filling masked or missing image areas with generated content, and example-based synthesis using retrieved visual samples.
The system includes capabilities for training both the regularized autoencoder models used for compression and the latent diffusion models used for synthesis. It supports various sampling methods, including unconditional image generation and classifier-free guidance to balance sample fidelity and diversity.
Features
- Latent Diffusion Models - Provides a latent diffusion model framework that performs the denoising process within a compressed latent space.
- Diffusion Model Training - Includes workflows for training latent diffusion models on large datasets of diverse visual examples.
- Latent Space Encoders - Implements latent space encoders to compress high-resolution images for efficient diffusion processing.
- Text-to-Image Generators - Generates high-resolution images from natural language descriptions using latent diffusion pipelines.
- Cross-Attention Conditioning - Uses cross-attention conditioning to integrate text and image embeddings into the denoising network.
- Variational Autoencoders - Uses variational autoencoders to compress high-resolution images into a lower-dimensional latent space.
- Iterative Denoising Pipelines - Implements iterative denoising pipelines to recursively remove Gaussian noise from random signals.
- Latent Noise Prediction - Performs iterative noise prediction within a compressed representation to improve computational efficiency.
- Classifier-Free Guidance - Implements classifier-free guidance to balance image fidelity and diversity during the denoising process.
- Image Inpainting - Provides image inpainting capabilities to restore missing areas by combining known pixels with generated content.
- Example-Based Synthesis - Implements example-based synthesis to create new visuals based on retrieved image patterns.
- Image Generation - Supports unconditional image generation by sampling directly from the trained model.
- Example-Based Synthesis - Creates new images by combining natural language prompts with similar visual examples retrieved from a database.
- High-Resolution Synthesis - Synthesizes high-fidelity, high-resolution images from trained generative models.
- Image Restoration and Generation - Provides image restoration and inpainting to fill masked areas and restore the visual whole.
- Area Filling and Clearing - Fills missing image areas and gaps with generated visual content.
- Latent Inpainting Masks - Supports mask-based latent inpainting to reconstruct missing image regions within the compressed space.
- Generation - Listed in the “Generation” section of the Awesome Diffusion Models awesome list.
- Others - Listed in the “Others” section of the Ailia Models awesome list.
- Text to Image - Listed in the “Text to image” section of the Ailia Models awesome list.
