jzhang38/TinyLlamaArchived
TinyLlama
TinyLlama is a compact 1.1B parameter language model pretrained on a dataset of 3 trillion tokens. It is an edge AI model designed for high-performance text generation on memory-constrained devices.
The project provides a distributed pretraining framework for training small language models across multiple GPUs and nodes. It also includes a finetuning toolkit for full-parameter weight adjustments to adapt the base model for chat and specific tasks.
The system supports distributed large language model training and on-device text generation. Its architectural components include rotary positional embeddings, root mean square layer normalization, and attention kernels.
Features
- Generative AI Models - Provides a lightweight 1.1B parameter generative model optimized for deployment on edge hardware.
- Data-Parallel Training - Provides a framework for distributing training workloads across multiple GPUs using data parallelism.
- Efficient LLM Development - Focuses on the development and optimization of compact models to enable high performance on constrained hardware.
- LLM Development Toolkits - Includes a toolkit for full-parameter weight adjustments to adapt the model for specific tasks.
- Large Language Model Training Frameworks - Executes distributed pretraining of transformer models across multi-GPU environments using optimized attention.
- Model Pretraining Frameworks - Provides a distributed system for training foundational language models across multiple GPUs and nodes.
- Finetuning Workflows - Provides workflows for adapting a pretrained base model to specific tasks or chat styles.
- Language Model Pretraining - Trains a compact Llama model on a 3 trillion token dataset using distributed GPU clusters.
- On-Device Text Generation Runners - Enables real-time text generation and dialogue execution on memory-constrained edge hardware.
- Language Models - Implements a compact 1.1B parameter language model for high-performance text generation.
- FlashAttention - Uses FlashAttention kernels to minimize GPU memory access and increase attention computation speed.
- Full Parameter Fine-Tuning - Ships scripts for updating all model parameters to adapt the general model for chat interactions.
- Causal Masking - Implements causal masking to prevent the model from attending to future tokens during training.
- RMSNorm Layers - Implements root mean square layer normalization to stabilize neural network activations during training.
- Rotary Positional Embeddings - Utilizes rotary positional embeddings to encode relative token positions in a high-dimensional space.
- Pretrained Weight Initializers - Employs specific weight initialization distributions to ensure stable convergence during large-scale pretraining.
- Large Language Models - Compact language models for efficient deployment.
- Small Language Models - Pre-trained small-scale LLaMA model.
- Large Language Models (LLMs) - Listed in the “Large Language Models (LLMs)” section of the The Incredible Pytorch awesome list.
