datawhalechina/llms-from-scratch-cn

Features

• Generative Models - Implements a complete pipeline for building generative language models from scratch, including tokenization and pre-training.
• Large Language Model Curricula - Offers a structured educational program for learning the internal mechanics of transformer-based language models.
• Causal - Implements causal masking to prevent the model from attending to future tokens during autoregressive generation.
• Attention Mechanisms - Implements scaled dot-product and multi-head attention to demonstrate how models weight token importance.
• Foundation Model Pre-training - Implements the process of training base language models from scratch on large unlabeled corpora.
• Language Model Pre-training - Implements the full pre-training process, including data preprocessing and training loops on raw text corpora.
• Large Language Models - Provides a comprehensive implementation of modern large-scale language model architectural components.
• Llama Architectures - Implements the structural configurations and transformer blocks specific to the Llama model family.
• LLM Implementations - Provides concrete software implementations of diverse LLM architectures like Llama and GLM.
• Byte Pair Encodings - Implements subword tokenization using the byte-pair encoding algorithm to handle text corpora.
• Multi-Head Attention Mechanisms - Implements multi-head attention mechanisms to capture diverse linguistic relationships in parallel subspaces.
• Transformer Blocks - Stacks attention mechanisms and feed-forward networks into repetitive transformer blocks for hierarchical representation.
• Pre-training Tutorials - Provides practical instructions for cleaning text and training models on unlabeled corpora.
• LLM Education - Serves as an educational resource for implementing large language models from the ground up.
• Transformer Implementations - Provides pedagogical, from-scratch implementations of transformer architectures using basic Python frameworks.
• Generative AI Prototyping - Provides a complete pipeline for prototyping functional text generation systems from scratch.
• Gradient Clipping - Provides training utilities that combine learning rate scheduling with gradient clipping for stable convergence.
• Model Architecture Analysis - Provides a framework to inspect model definitions and configurations to analyze the structural mechanisms of different model families.
• Model Architecture Comparisons - Implements and compares diverse model families like Llama, GLM, and RWKV to analyze structural differences.
• Sliding-Window Data Segmentation - Generates overlapping input and target sequence pairs from continuous text using a sliding window.
• Text Preprocessing Pipelines - Implements an automated pipeline for cleaning and formatting raw text data before model consumption.
• Training Dataset Preparation - Provides a process for tokenizing raw text and preparing overlapping sequences for model training.
• Mathematical Equivalence Analysis - Demonstrates the mathematical equivalence between embedding layers and fully connected layers using one-hot encoded vectors.
• Configuration-Driven Assembly - Uses external configuration files to instantiate and assemble different model architectures from shared components.
• Courses and Tutorials - Chinese course on building LLMs from scratch.
• Learning Resources - Chinese-language guide to building LLMs.

Star history