lucidrains/vector-quantize-pytorch

Features

• Differentiable Quantization - A set of tools for estimating gradients through non-differentiable quantization steps using straight-through estimators and rotation tricks.
• Vector Quantization - Implements core vector quantization mechanisms for mapping continuous latent vectors to discrete codebook entries.
• Discrete Latent Representations - Converts continuous vectors into discrete codes to create categorical representations for machine learning.
• Finite Scalar Quantization - Rounds each scalar into discrete levels to create a hypercube of codes and eliminate commitment losses.
• Vector-Quantized VAEs - Supports building VQ-VAE architectures by mapping continuous latent spaces to a learnable codebook.
• Representational Collapse Prevention - Prevents codebook collapse by ensuring inactive vectors are re-initialized to maintain high utilization.
• Stochastic Gradient Estimators - Estimates gradients through non-differentiable quantization steps using straight-through estimators to enable backpropagation.
• Quantization Gradient Estimators - Uses advanced estimators like the rotation trick to calculate gradients through non-differentiable quantization steps.
• Straight-Through Estimators - Provides straight-through gradient estimation to enable backpropagation through non-differentiable quantization steps.
• Residual Vector Quantizers - Implements residual vector quantizers that use multiple additive codebooks to refine representation precision.
• Vector Quantization - Converts continuous latent vectors into discrete codebook entries using distance-based matching.
• Scalar Quantization - Provides finite scalar quantization by rounding dimensions to discrete levels, creating hypercube codes.
• Codebook Collapse Prevention - Prevents codebook collapse by replacing unused vectors with random samples from the current training batch.
• Codebook Management - Provides tools for maintaining high utilization of embedding vectors and pruning inactive entries during training.
• Multi-Head Quantization - Supports multi-head parallel quantization to divide input dimensions across independent codebooks.
• Codebook Toolkits - Ships a toolkit for initializing codebooks with K-Means and preventing collapse via random vector replacement.
• Disentanglement - Uses scalar quantization and binary latents to separate distinct data features within the latent space.
• Latent Space Disentanglement - Assigns discrete code vectors using learnable scalar codebooks for each dimension to capture independent data features.
• Implicit Quantization Code Generation - Produces discrete codes through linear projections using frozen codebooks to improve convergence and prevent collapse.
• Codebook Initialization - Implements k-means centroid initialization to accelerate convergence by starting codebooks at data centroids.
• Codebook Initializations - Sets initial codebook vectors using k-means centroids calculated from the first data batch to accelerate convergence.
• Latent Scalar Quantization - Assigns discrete code vectors using a learnable scalar codebook for each dimension to improve feature separation.
• Lookup-Free Quantization - Discretizes inputs using independent binary latents to remove the requirement for an embedding lookup table.
• Multi-Head Quantization - Quantizes input dimensions across multiple heads using either shared or separate codebooks for each head.
• Angular Distance Matching - Implements angular distance matching using L2 normalization and cosine similarity for codebook lookups.
• Finite Scalar Quantization - Implements a finite scalar quantizer that rounds scalars into discrete levels to create hypercube codes.

Star history