lucidrains/x-transformers

Features

• Experimental Transformer Libraries - Serves as a modular PyTorch library for implementing and testing experimental transformer architectures.
• Transformer Architectures - Provides a modular framework for constructing various transformer architectures for research tasks.
• Advanced Attention Research - Integrates specialized mechanisms like Flash Attention and talking heads for advanced attention research.
• Tiled Attention Kernels - Ships high-performance tiled-kernel attention implementations to minimize memory usage and increase speed.
• Dynamic Positional Biases - Uses relative position-based biases that generalize to sequence lengths longer than those seen during training.
• Flash-Attention Implementations - Provides high-performance Flash-Attention implementations to reduce memory access and compute for long sequences.
• Relative Position Bias Computations - Adds learned bias values to the attention matrix to provide flexible relative positional encoding.
• Attention Optimization Suites - Provides a comprehensive suite of memory-efficient attention mechanisms including Flash Attention and multi-query attention.
• Autoregressive Models - Provides decoder-based architectures with KV caching for efficient autoregressive sequence generation.
• Cross-Attention Implementations - Implements cross-attention mechanisms allowing one transformer to attend to the encoded output of another.
• Decoder Architectures - Creates modular transformer-based decoder architectures for autoregressive sequence generation.
• Long-Context Models - Extends the context window of transformers using recurrence and relative positional biases.
• GPT Decoder Construction - Implements patterns for constructing autoregressive GPT-style decoders with customizable depth and attention heads.
• Recurrence Caching Mechanisms - Provides KV-cache based recurrence mechanisms to extend the effective context window of models.
• Modular Layer Compositions - Provides a modular framework for constructing transformer architectures by assembling interchangeable attention and feedforward blocks.
• Autoregressive Model Interfaces - Provides wrappers and interfaces to support autoregressive tasks and custom meta-learning loss functions.
• Multi-Query Attention - Implements multi-query attention sharing to reduce memory overhead during inference.
• Gated Feedforward Layers - Implements gated-linear-unit feedforward layers to enhance non-linear representation in transformers.
• Gating Configurations - Integrates GLU and Swish GLU variants into feedforward layers to enhance model representation accuracy.
• Linear Bias Positional Embeddings - Implements static linear biases in the attention matrix to enable length extrapolation for autoregressive models.
• Rotary Positional Embeddings - Integrates rotary positional embeddings to inject relative position information via vector rotations.
• Sequence Generation - Provides general sequence generation capabilities for producing tokens and numerical values iteratively.
• Autoregressive Text Generation - Implements autoregressive token generation for text sequences using decoder architectures and key-value caching.
• Text Sequence Generation - Produces new natural language text sequences by predicting subsequent tokens from a prompt.
• Positional Encodings - Provides relative positional encoding using learned bias values as an alternative to absolute embeddings.
• Transformer Encoder Architectures - Provides a modular architecture for processing sequences of data into dense vector representations.
• Transformer Sequence Modeling Toolkits - Offers a toolkit for extending context windows using recurrence and rotary embeddings.
• Vision Transformers - Implements transformer wrappers for image processing using patch-based inputs for classification and captioning.
• Transformer Model Construction - Constructs modular encoder, decoder, or full transformer architectures with configurable depth and dimensions.
• Neural Token Sequence Generation - Generates a series of subsequent tokens from initial neural embeddings for a specified number of steps.
• Advanced Normalization Schemes - Implements advanced normalization schemes like RMSNorm and Sandwich Norm to stabilize training.
• Gated Attention Outputs - Applies a gated linear unit to the output of the attention layer based on original queries.
• Learned Memory Augmentations - Integrates learned memory pairs or dedicated register tokens to alleviate attention outliers and improve performance.
• Recurrent-Attention Hybrids - Combines attention mechanisms with recurrent neural networks to track state across long sequences.
• Residual Scoring - Adds pre-attention scores across all layers to improve model performance without adding new parameters.
• Continuous Embedding Processing - Processes continuous-valued input and output tensors as an alternative to discrete token identifiers.
• Latent Autoregressive Modeling - Predicts next-token embeddings using compressed latent representations and specialized loss functions.
• Input Token Shifting - Shifts subsets of the feature dimension along the sequence dimension to accelerate training convergence.
• KV Cache Management - Provides KV-cache-based recurrence to efficiently store and retrieve key-value pairs for long sequences.
• Learnable Memory Tokens - Integrates learned register tokens into attention layers to stabilize training and reduce outliers.
• Hybrid Normalization Placements - Combines pre- and post-layer normalization to balance stability and representation power.
• Sandwich Normalizations - Adds an extra layer normalization to all branch outputs to resolve training instability.
• Causal Masking - Combines masked language modeling with autoregressive training by randomly masking tokens.
• Causal Masked Modeling Objectives - Combines masking objectives from masked language modeling with autoregressive training for improved results.
• Sublayer Reordering - Adjusts the sequence of attention and feedforward blocks to optimize model depth and performance.
• Latent-Variable Autoregressive Models - Includes model templates that combine autoregressive generation with variational autoencoders to condition outputs on latent spaces.
• Attention Sparsification - Implements top-k selection to zero out low-importance attention scores, reducing computational overhead.
• GPT-VAE Hybrids - Creates variational autoencoders that combine autoregressive generation with a latent space.
• Attention Head Mixing - Linearly combines information across different attention heads before and after the softmax operation.
• Talking Heads Attention - Implements Talking Heads Attention by mixing information between heads via linear combinations around the softmax.
• Neural Network Stability Optimizations - Implements advanced normalization schemes like RMSNorm and Sandwich Norm to stabilize training.
• Normalization Layers - Uses combined pre- and post-layer normalization to reduce representation collapse while maintaining stability.
• Hybrid Normalization Schemes - Combines multiple normalization types, including RMSNorm and Sandwich Norm, to prevent gradient collapse.
• Numerical Encoding Schemes - Encodes numerical data into continuously scaled tokens to improve generalization for arithmetic and forecasting.
• Persistent Memory Injection - Injects learned key-value pairs into the attention mechanism to improve feedforward layer performance.
• PyTorch Vision Transformer Frameworks - Provides a PyTorch-based framework for building patch-based Vision Transformers.
• Query-Key Interaction Normalizations - L2-normalizes queries and keys before the dot product to prevent overflow and stabilize training.
• Gated Residual Connections - Integrates gating mechanisms into residual connections to improve transformer network stability and performance.
• Segment-Level Recurrence - Implements segment-level recurrence to cache hidden states from previous segments, extending the effective context length.
• Sequence Length Extrapolation - Enables models to process sequences longer than the training window using specialized positional embeddings.
• Macaron Architectures - Implements transformer configurations that sandwich an attention layer between two feedforward layers.
• Transformer Layer Topologies - Allows rearranging sublayers using custom execution orders or coefficients to optimize information flow.
• Weight Tying Strategies - Shares weights across different transformer layers to reduce the total parameter count.
• Transformer Implementations - Full-attention transformer implementation with experimental research features.

Star history