NVlabs/SPADE
SPADE
SPADE is a semantic image synthesis framework and generative adversarial network designed to transform semantic label maps into photorealistic images. It uses a spatially-adaptive normalization model to modulate activations based on semantic maps, ensuring that spatial layouts and details are preserved throughout the synthesis process.
The project enables the generation of diverse image variations from a single semantic layout by integrating variational autoencoders and latent vector style control. These mechanisms allow for the adjustment of visual appearances and textures while keeping the underlying structural layout constant.
The system includes workflows for training synthesis models on custom image and label pairs and supports reverse mapping training to convert photos back into semantic maps. Training is supported by multi-GPU distributed acceleration to handle high-resolution image generation.
Features
- Spatially-Adaptive Normalization - Implements a spatially-adaptive normalization model that preserves layout details by modulating activations based on semantic maps.
- Adaptive Instance Normalization - Implements an adaptive normalization model that modulates activations based on semantic maps to maintain layout details.
- Image-to-Image Translation - Performs image-to-image translation by mapping semantic layout domains into photorealistic image domains.
- Generative Adversarial Architectures - Uses a generative adversarial network architecture to synthesize diverse visual styles from semantic layouts.
- Generative Adversarial Image Synthesis - Generates photorealistic images using a GAN-based architecture guided by semantic label maps.
- Semantic Synthesis Frameworks - Provides a deep learning system that transforms semantic label maps into photorealistic images using spatially-adaptive normalization.
- Latent Style Control - Adjusts the visual appearance and textures of generated images using latent vectors while keeping the structural layout constant.
- Spatially-Aware Layers - Replaces standard normalization with learned tensors that adjust feature maps based on local semantic class labels.
- Variational Autoencoders - Integrates variational autoencoders to encode visual styles into latent vectors for diverse image synthesis.
- Structural Image Generation - Converts structural semantic maps into detailed photorealistic images while preserving the spatial arrangement of regions.
- Visual Guidance Inputs - Uses semantic label maps as visual guidance to direct the placement and structure of generated objects.
- Semantic Image Synthesis - Generates photorealistic images from semantic label maps to create visuals that match a specific layout.
- Segmentation Model Training - Trains models to perform the inverse task of converting existing photorealistic images back into semantic label maps.
- Generative Latent Mappings - Adjusts the visual appearance of results by mapping noise/latent vectors to high-dimensional representations for synthesis.
- Image Variation and Mixing - Creates multiple distinct visual versions of a single semantic layout using latent space variations.
- Synthesis Dataset Training - Supports training neural networks on custom image and label pairs to learn realistic scene synthesis.
- Semantic-to-Image Converters - Renders high-resolution photos from segmentation maps using trained synthesis networks.
- Foundational Generative Models - Official implementation for semantic image synthesis.
- Generative Models - Spatially-adaptive normalization for semantic image synthesis.
