datawhalechina/so-large-lm

Features

• Large Language Model Curricula - Provides a structured educational program for learning the engineering and application of large language models.
• Autonomous Agent Designers - Provides design patterns and architectural frameworks for building autonomous systems that use LLMs as reasoning engines.
• Agentic Reasoning Frameworks - Outlines architectural patterns that integrate internal reasoning loops with external tool execution.
• AI Ethics and Fairness - Examines social bias, hallucinations, copyright law, and ethical considerations in the development of LLMs.
• LLM Safety and Ethics - Includes a dedicated educational module examining social bias, hallucination, copyright, and environmental impact.
• Attention Computations - Details the fundamental computation of matching queries against key-value pairs to produce weighted token combinations.
• Adapter Layers - Explains the use of lightweight trainable adapter layers to specialize frozen pretrained models for specific tasks.
• Bottleneck Layers - Covers the implementation of bottleneck residual networks for efficient model adaptation.
• Search-Based Reasoning Strategies - Explores multiple reasoning paths using search strategies to select the most promising route for problem solving.
• Chain-of-Thought Prompting - Covers chain-of-thought prompting techniques to enable complex reasoning through structured intermediate steps.
• Attention Weight Injection - Teaches the implementation of learnable attention weights to capture structural dependencies.
• Retrieval-Augmented Generation - Teaches how to ground language model responses by retrieving relevant documents from a search index.
• Decoder Architectures - Explains transformer structures utilizing causal attention mechanisms for autoregressive sequence generation.
• Distributed Training - Teaches data and model parallelism strategies to train large neural networks across multiple devices.
• Prompt Embedding Adaptation - Provides a guide on tuning model behavior by prepending learnable continuous token embeddings.
• Encoder-Decoder Architectures - Covers neural network designs that map input sequences to output sequences via intermediate representations.
• External Tool Integrations - Explains how to connect AI assistants to external utilities like search APIs and browser automation tools.
• Full Parameter Fine-Tuning - Guides users through full-parameter fine-tuning workflows to optimize core weights and task-specific heads.
• LLM Application Development - Guides the development of autonomous agents and retrieval-augmented generation systems using language models.
• Human Preference Alignment - Provides resources on using human feedback and reinforcement learning to align model outputs with safety guidelines.
• Byte Pair Encodings - Covers subword tokenization using iterative character pair merging to build model vocabularies.
• Multi-Head Attention Mechanisms - Provides detailed instruction on running parallel attention heads to capture diverse dependencies within text.
• Text Tokenization - Includes guides on transforming raw strings into sequences using BPE or Unigram tokenization models.
• Contextual Embeddings - Explains how transformer architectures map token sequences into vectors that capture meaning based on surrounding context.
• Parameter Efficient Fine-Tuning - Offers comprehensive tutorials on memory-efficient fine-tuning by updating only a small subset of model parameters.
• Prefix Vector Tuning - Explains how to use learnable prefix vectors in attention layers for task-specific adaptation.
• Prompt Embedding Tuning - Details how to optimize model inputs using learnable continuous token embeddings.
• Reinforcement Learning Alignment - Details reinforcement learning alignment techniques to reduce toxicity and improve human value adherence.
• Representation Probing - Provides tutorials on using lightweight prediction layers to probe learned features in frozen models.
• Self-Attention Mechanisms - Explains the mechanism where each token in a sequence is compared to all others to exchange information.
• Dynamic Plan Refinement - Details processes for adjusting task subgoals and action sequences based on real-time environmental feedback.
• Encoder-Only Classification Methods - Details bidirectional text classification using contextual embeddings for tasks like sentiment analysis.
• Token Prediction - Details the specific token-level prediction mechanisms used to generate text sequences.
• Transformer Architectures - Provides detailed study of transformer internals, including attention mechanisms, tokenization, and decoder architectures.
• LLM Training and Optimization - Manages large-scale model development including distributed training and parameter-efficient fine-tuning.
• Transformer Representation Analysis - Explains how to analyze internal transformer representations using linear and feed-forward probes.
• Agent Memory Management - Teaches systems for storing and retrieving long-term memories and execution traces to maintain interaction continuity.
• LLM Training Modules - Provides instructional modules on data preparation, tokenization, and distributed training strategies for large models.
• LLM Education - Provides a comprehensive educational curriculum covering the theory, architecture, training, and deployment of large language models.
• Transformer Architecture Walkthroughs - Ships a structured tutorial explaining transformer internals, attention mechanisms, and decoder-only architectures.
• Encoder-Decoder Training Methods - Explains training methods for sequence-to-sequence encoder-decoder architectures.
• Attention Mechanisms - Describes the mathematical implementation of the query-key-value attention mechanism.
• Text Dataset Curators - Provides guidance on pipelines that filter, format, and deduplicate text data to create high-quality LLM training datasets.
• Representational Harm Auditors - Guides the analysis of training data for representational harms and demographic imbalances.
• Environmental Impact Assessment - Includes tools and methods for quantifying the energy and carbon footprint of machine learning workloads.
• Feed-Forward Network Layers - Describes the two-layer nonlinear transformations used to add representational capacity to each token.
• One-Shot Learning - Explains machine learning techniques for executing tasks based on a single training example.
• Few-Shot Text Learning - Covers techniques for text classification and entity extraction using minimal labeled examples in prompts.
• Reliability Optimizations - Explains how to implement automated error detection and scalable storage to minimize downtime during long training runs.
• Toxicity Evaluation Metrics - Provides metrics for measuring the probability of toxic completions generated from specific prompts.
• Multi-Node Training Scaling - Provides techniques for distributing model and data across multiple hardware nodes to scale training.
• Data Parallelism - Explains how to split datasets across multiple devices and synchronize gradients using AllReduce.
• Learning Rate Schedulers - Describes algorithms for dynamically adjusting learning rates and momentum to improve convergence.
• LLM Operational Cost Optimization - Provides strategies to minimize financial and environmental expenses associated with training and running models.
• LLM Training Cost Estimation - Teaches how to calculate financial and energy costs associated with training and running large models.
• Logical and Arithmetic Reasoning - Covers the capability of models to solve non-linguistic logic and arithmetic problems.
• Long-Form Text Generation - Teaches the capability to produce extended, coherent sequences of natural language text.
• Perplexity Calculators - Includes utilities for calculating model perplexity using cross-entropy loss to assess predictive quality.
• Sequential Text Processing Pipelines - Teaches how recursive hidden states are computed using RNN, LSTM, or GRU cells for sequential text processing.
• Mixed Precision Training - Covers techniques for using lower-bit precision formats to accelerate training and reduce memory usage.
• Mixture of Experts - Implements routing and recording of expert paths within Mixture-of-Experts architectures.
• Expert Load Balancers - Explains strategies for routing traffic across model experts to prevent computational bottlenecks.
• Multi-Stage Pipelines - Explains the progression from self-supervised pretraining through supervised fine-tuning to reinforcement learning from human feedback.
• Model Parallelism - Covers strategies for partitioning model parameters across multiple GPUs to exceed single-device memory limits.
• Masked Language Modeling - Explains pre-training techniques using masked language modeling to build bidirectional representations.
• Model Auditing Tools - Provides a framework for auditing model performance disparities across different demographic groups.
• Decoder-Only Training Methods - Covers training procedures for autoregressive decoder-only models using causal left-to-right context.
• Encoder-Only Training Methods - Provides guides on training bidirectional encoder-only models using masked language modeling.
• N-Gram Language Models - Explains statistical n-gram models for predicting word sequences.
• Long-Range Dependency Mechanisms - Covers techniques for capturing dependencies between distant tokens to overcome the limitations of n-gram models.
• Stabilization Techniques - Covers the use of residual connections and layer normalization to ensure stable neural network training.
• Pipeline Parallelisms - Describes the strategy of splitting a neural network into sequential stages assigned to different devices.
• Adam Optimizers - Provides educational material on Adam optimizers for stable gradient-based optimization.
• Sinusoidal Encodings - Explains the use of fixed sine and cosine functions to encode sequence order in transformer models.
• Prompt-Based Language Switching - Provides guidance on changing output languages via instructions within the prompt context.
• Prompt-Based Text Generation - Covers generating text sequences conditioned on input prompts using generative models.
• Retrieval Augmented Generation Pipelines - Describes the end-to-end pipeline for integrating external data retrieval with language model generation.
• Sequence-to-Sequence Transformer Architectures - Explains integrated transformer architectures that combine encoder and decoder components for translation.
• Supervised Fine-Tuning - Provides tutorials on adapting pre-trained models using labeled instruction datasets for specific tasks.
• Task Decompositions - Explains methods for breaking down complex objectives into a sequence of executable sub-tasks.
• Decoding-Based Toxicity Reducers - Details techniques for reducing toxic output via data-based training and decoding guidance.
• Source Documenters - Offers resources on documenting the origin, composition, and curation process of datasets used for model training.
• Training Memory Management - Teaches utilities for optimizing memory usage and handling hardware constraints during large-scale training.
• Transformer Blocks - Explains the interaction between self-attention and feed-forward layers within a transformer block.
• Weight Initialization - Describes methods for setting initial neural network parameter values to improve stability.
• Zero-Shot Generalization Tuning - Covers strategies for improving zero-shot performance through diverse prompt format fine-tuning.
• Probability-Based Bias Detection - Covers the detection of social bias by analyzing probability differences across demographic groups.
• Conditional Generation Tasks - Demonstrates the use of conditional generation for tasks like question answering and article creation.
• Natural Language Understanding - Covers the use of encoder-only architectures for generating contextual embeddings used in classification tasks.
• Poisoning Detections - Covers the identification and mitigation of malicious training data used to corrupt model behavior.
• From-Scratch Training - Teaches the process of training foundational models from scratch without pre-existing weights.
• Question Answering - Explains prompting strategies for generating answers using only the model's internal knowledge.
• Training Carbon Footprint Estimation - Provides methods to calculate the carbon footprint of training based on hardware, runtime, and energy sources.
• K-Nearest Neighbor Retrieval - Provides educational material on k-nearest neighbor retrieval for identifying similar sequences.
• Optimizer State Compression - Details the use of AdaFactor to reduce training memory via low-rank momentum approximations.
• AI Risk Assessments - Provides processes for evaluating the security posture and misinformation risks of AI systems.
• Content Moderation - Covers the classification of user text as harmful or acceptable using trained moderation models.
• Disinformation Detection - Teaches how to identify false or misleading content by analyzing deceptive intent in text.
• Few-Shot Harm Detection Models - Covers identifying hate speech and incitement to violence using few-shot learning models.
• Lifecycle Policies - Establishes policies for the responsible creation, maintenance, and use of datasets throughout their entire lifecycle.
• LLM Safety Enforcers - Provides resources for evaluating and enforcing controls against jailbreaks, PII leaks, and hallucinations.
• Energy Consumption Analyzers - Provides utilities for measuring and reporting the power usage and carbon footprint of software workloads.
• Dataset Contamination Detection - Teaches methods for identifying overlap between training data and evaluation benchmarks to ensure valid performance metrics.
• Naturalness Scoring - Explains how to use probability scores for token sequences to evaluate the naturalness of generated text.
• AI and Machine Learning - Deep dives into the architecture of massive language models.

Star history