naklecha/llama3-from-scratch

Features

• Large Language Models - Implements a Llama 3 transformer architecture from scratch using fundamental tensor operations.
• Transformer Language Models - Provides a manual reconstruction of the transformer-based language model architecture using PyTorch.
• Attention Mechanisms - Calculates relationship scores between tokens using query and key matrix multiplication.
• Neural Network Operations - Manages mathematical data flow through rotary embeddings, gated linear units, and normalization layers.
• Multi-Head Attention Mechanisms - Implements multi-head attention to compute contextual relationships across parallel attention heads.
• Neural Network Implementations - Implements a PyTorch-based neural network covering tokenization, rotary embeddings, and multi-head attention.
• Rotary Positional Embeddings - Applies rotational shifts to query and key vectors to encode relative token positions.
• Token Prediction - Predicts the most probable next token by calculating a probability distribution over the vocabulary.
• Transformer Blocks - Constructs the model using sequential layers of attention and feed-forward networks via matrix operations.
• Vector Embeddings - Maps discrete numerical tokens to high-dimensional continuous vectors to represent semantic meaning.
• LLM - Offers a manual reconstruction of the Llama 3 architecture using basic matrix operations.
• Gated Linear Units - Implements gated linear units to introduce non-linearity within the feed-forward network.
• Normalization Layers - Stabilizes numerical computations using root mean square normalization layers.
• Model Inference - Provides a system to load model weights and generate text predictions during inference.
• Model Weight Management - Includes mechanisms to load pre-trained tensor weights and configuration parameters to initialize the network.
• Neural Network Layers - Implements a SwiGLU feed-forward network using gated linear units for non-linear data processing.
• RMS Normalizations - Stabilizes neural network activations by scaling tensors based on the root mean square of their elements.
• Transformer Architectures - Serves as a reference for studying the internal mechanics of transformer architectures through manual implementation.
• LLM - Provides a detailed reference for the internal mathematical structure and tensor flow of Llama 3.
• LLM Development and Research - Implementation of a language model from basic matrix operations.
• Educational Resources - Educational project demonstrating model architecture from the ground up.

Star history