openlm-research/open_llama
Open Llama
Open Llama is an open source large language model and pre-trained transformer designed as a permissively licensed alternative to proprietary weights. It serves as a base model reproduction of the Llama architecture, providing a set of weights for a decoder-only transformer.
The project provides a transparently trained model based on the RedPajama dataset, supporting unrestricted commercial and research use. It includes systems for serving pre-trained weights in various sizes.
The project covers natural language processing research and performance benchmarking through text quality evaluation harnesses. It is designed for integration with deep learning frameworks and research pipelines.
Features
- Transformer Architecture Implementation - Provides a complete implementation of the Llama architecture as a permissively licensed alternative to proprietary weights.
- Large Language Models - Ships a pre-trained base model designed for downstream research and fine-tuning applications.
- Transformer Architectures - Implements a transformer-based architecture using stacked self-attention layers and feed-forward networks.
- Natural Language Processing - Serves as a reproducible baseline for developing and testing new natural language processing techniques.
- Causal Language Modeling - Implements the architectural objective of predicting the next token in a sequence by masking future tokens.
- Data Curation - Utilizes the RedPajama dataset through rigorous filtering and refinement to ensure a high-quality training corpus.
- Open Source Models - Provides a transparently trained model that allows for unrestricted commercial and research use.
- Pre-trained Models - Supplies a set of pre-trained weights for a decoder-only transformer based on large-scale data.
- Deep Learning Framework Implementations - Ensures compatibility between model implementations and industry-standard libraries like PyTorch and JAX.
- Deep Learning Pipelines - Provides compatibility with deep learning pipelines to integrate the model into existing research workflows.
- Text - Provides standardized harnesses to measure the accuracy and performance of generated text.
- LLM Benchmarking - Includes standardized evaluation harnesses to measure text generation quality and accuracy against other models.
- Rotary Positional Embeddings - Uses rotary positional embeddings to encode relative token positions through vector rotation in a complex plane.
- RMS Normalizations - Employs root mean square normalization to stabilize internal activations and improve training convergence.
- KV Cache Management - Includes strategies for optimizing the key-value cache to avoid redundant computations during text generation.
- Large Language Model Deployments - Includes systems for serving pre-trained weights in various sizes for deployment on private hardware.
- Natural Language Processing - Listed in the “Natural Language Processing” section of the FunNLP awesome list.
- Large Language Models (LLMs) - Listed in the “Large Language Models (LLMs)” section of the The Incredible Pytorch awesome list.
