harvardnlp/annotated-transformer
Annotated Transformer
The Annotated Transformer is an educational resource that provides annotated code implementations of the Transformer architecture for sequence-to-sequence tasks, built with PyTorch. It serves as a learning tool for understanding attention mechanisms, multi-head parallel attention, and scaled dot-product attention through executable examples that walk through each component of the model.
The project covers the full Transformer pipeline, including stacked encoder-decoder layers with residual connections and layer normalization, sinusoidal positional encoding for order-aware representation, and masked self-attention decoding for auto-regressive generation. It also demonstrates label smoothing regularization to reduce overconfidence during training, and provides a framework for neural machine translation that encodes input sequences and decodes output sequences using attention mechanisms.
The documentation includes annotated code that explains how multi-head self-attention works, how positional encoding injects sequence order information, and how the model processes sequences through stacked self-attention and feed-forward layers. The resource is designed to help learners implement and train Transformer-based sequence models for translation tasks, with explanations of each architectural component from the original paper.
Features
- Encoder-Decoder Architectures - Processes input through identical encoder layers and generates output via decoder layers with cross-attention.
- Sequence Encoders - Processes an input sequence through stacked self-attention and feed-forward layers to produce continuous representations.
- Causal Masking - Implements causal masking to prevent future token leakage during auto-regressive sequence generation.
- Multi-Head Attention Mechanisms - Splits queries, keys, and values across multiple parallel heads to learn distinct representation subspaces.
- Sinusoidal Encodings - Injects fixed sinusoidal signals into token embeddings to encode absolute and relative position information.
- Residual Connection Implementations - Adds skip connections around each sub-layer followed by layer normalization to stabilize deep training.
- Sequence Decoders - Generates an output sequence token by token using masked self-attention and encoder-decoder attention.
- Sequence Learning Models - Builds models that transform input sequences into output sequences using attention mechanisms.
- Transformer Architecture Implementation - Implements the full Transformer architecture from the original paper for sequence-to-sequence tasks.
- Positional Encodings - Injects sinusoidal signals into token embeddings to provide the model with information about token order.
- Paper Implementations - Provides an annotated code implementation of the Transformer architecture with explanations of each component.
- Scaled Attention Computations - Computes attention by scaling query-key dot products before softmax to prevent gradient vanishing in high-dimensional spaces.
- Educational Tutorials - Provides annotated code examples that teach how multi-head self-attention and scaled dot-product attention work.
- Neural Machine Translation - Trains sequence models with label smoothing and positional encoding for translation tasks.
- Neural Machine Translation Frameworks - Provides a framework for encoding input sequences and decoding output sequences using transformer layers.
- Label Smoothing Techniques - Demonstrates label smoothing regularization to reduce overconfidence during transformer training.
- PyTorch Deep Learning Examples - Ships a deep learning model built with PyTorch for processing sequential data using attention mechanisms.
Star history
How this analysis was created: This summary and feature list were written by an AI model that read the project's README and public documentation pages. Each feature links to the documentation it came from; stars, license and language come straight from the GitHub API. The model does not read the source code, and the analysis is refreshed when the project is re-analysed. Learn more on our About page.
