soumith/ganhacks
Ganhacks
This project is a PyTorch-based generative framework and implementation template for building Generative Adversarial Networks. It provides a collection of foundational toolkits and architectural patterns designed to synthesize high-quality artificial data while focusing on the stability of adversarial neural networks.
The framework distinguishes itself through a specialized toolkit for conditional image generation, which integrates discrete labels and auxiliary classification into the training process. It utilizes specific mechanisms to guide the generative process toward target classes by converting embeddings into image channels and using auxiliary labels for simultaneous authenticity detection and classification.
Broadly, the project covers adversarial model stabilization and training optimization to prevent common failure modes like mode collapse and vanishing gradients. This includes capabilities for gradient flow maintenance, latent space sampling, and training health monitoring via the tracking of gradient norms and loss variance.
The codebase implements a variety of training heuristics, including experience replay buffers, label smoothing, and adaptive optimizer pairings.
Features
- Generative Adversarial Networks - Implements and stabilizes GAN architectures to generate synthetic data while preventing mode collapse.
- Adversarial Loss Functions - Adjusts objective functions and implements label flipping to prevent vanishing gradients during generator training.
- Sampling - Implements Gaussian distributions and spherical interpolations for sampling latent space inputs.
- Conditional Training - Integrates discrete labels and auxiliary classification into the generative adversarial training process.
- Conditional Image Generation - Uses discrete labels and embeddings to guide a generative model toward producing specific categories of synthetic imagery.
- Model Implementation Templates - Provides a starter framework for building GANs based on researched training heuristics and architectural patterns.
- Adversarial Training Procedures - Implements competitive training loops for optimizing generator and discriminator networks in adversarial architectures.
- Adversarial Training Starters - Provides a collection of techniques and layers designed to stabilize training and prevent mode collapse.
- PyTorch Training Frameworks - Implements Gaussian latent sampling and normalization layers within a PyTorch-based generative framework.
- Generative Mode Collapse Prevention - Implements architectural and training heuristics to prevent vanishing gradients and output repetition.
- Synthetic Data Generators - Creates high-quality artificial samples through latent space sampling and discriminator feedback.
- Training Stability Techniques - Applies normalization and noise injection to prevent vanishing gradients and oscillations during competitive training.
- Conditional Embedding Layers - Utilizes embedding layers and additional image channels to support guided generation using discrete labels.
- Batch Normalization - Implements normalization using separate mini-batches for real and fake data to stabilize learning.
- Conditional Embedding Channels - Converts discrete labels into image channels to guide the generative process toward specific target classes.
- Auxiliary Classification Heads - Provides a classification head to the discriminator to improve training signals using ground truth labels.
- Persistent Noise Dropout - Implements high-rate dropout during both training and inference to provide continuous noise to the generator.
- Experience Replay Buffers - Implements memory structures that store historical generated samples to stabilize training and prevent catastrophic forgetting.
- Normalization Layers - Implements batch and instance normalization layers to center and scale activations for improved convergence.
- Gradient Flow Stabilizers - Uses leaky activations and avoids max-pooling to maintain stable gradient flow across deep network layers.
- Hyperparameter Tuning - Optimizes loss functions and optimizer pairings to improve convergence speed and training health.
- Hybrid Loss and Architecture Integration - Enables merging different model types or loss functions to stabilize training when standard architectures fail.
- Divergent Optimizer Pairings - Pairs different optimization algorithms for the generator and discriminator to balance the speed of convergence.
- Stochastic Noise Injection - Adds decaying Gaussian noise to inputs and layers to stabilize the adversarial learning process.
- Training Convergence Optimization - Accelerates training convergence by pairing adaptive ADAM optimizers for the generator and SGD for the discriminator.
- Simultaneous Authenticity and Class Labeling - Allows training a discriminator to perform simultaneous classification and authenticity detection.
- Label Smoothing Utilities - Provides utilities to adjust target labels with random noise to prevent discriminator overconfidence.
- Generative Adversarial Networks - Practical tips and tricks for training stable generative models.
- Educational Resources - Collection of tips and tricks for training GANs.
