facebookresearch/llama
Llama
Llama is a large language model runtime and inference engine designed to load and execute autoregressive transformer models. It enables the generation of natural language text completions from prompts using pretrained weights.
The system features multi-GPU model parallelism, which distributes model weights and workloads across multiple graphics processors to support larger parameter counts. It also incorporates a content safety filter that uses classifiers to intercept and block unsafe inputs or outputs during the inference process.
The project covers broad capabilities in distributed model execution, GPU resource scaling, and AI safety filtering.
Features
- Generative Text Inference - Provides a system for producing natural language text completions from prompts using large language models.
- Distributed Model Execution - Spreads large model workloads across multiple graphics processors to handle high parameter counts.
- Hardware Acceleration Kernels - Offloads heavy tensor mathematical operations to specialized GPU kernels for high-throughput processing.
- Inference Execution Engines - Acts as a runtime for executing large language models to generate text completions from prompts.
- Large Language Models - Runs pretrained Llama models to generate natural language text completions based on provided prompts.
- Autoregressive Decoding Strategies - Generates text one token at a time by feeding previous outputs back into the model.
- Large Language Model Runtimes - Provides a technical implementation for loading and running autoregressive transformer models on compatible hardware.
- Sharded Device Mapping - Maps specific model layers to individual graphics processors to support models exceeding single GPU memory.
- Distributed Execution - Implements distributed execution of model workloads across multiple GPUs to handle high parameter counts.
- Multi-GPU Distribution - Distributes model weights and workloads across multiple graphics processors to handle large parameter counts.
- Weight Distribution - Distributes model weights across multiple graphics processors to support models larger than a single GPU's memory.
- Tensor Parallelism - Splits large weight matrices across multiple GPUs to perform matrix multiplications in parallel.
- KV Cache Management - Manages a key-value cache in memory to avoid redundant computations during autoregressive generation.
- GPU Resource Scaling - Manages model parallelism to distribute weights across available hardware for efficient large scale deployment.
- Content Filtering - Uses separate safety models to detect and block prohibited content from reaching the user.
- Model Inference Filtering - Intercepts and blocks unsafe inputs or outputs during the inference process using secondary classifiers.
- Model Safety Filters - Implements a set of classifiers that intercept and block unsafe inputs or outputs during inference.
- Decoder Models - Efficient foundation language model for text generation.
- Foundation Models - Open-source large language model series.
- General Purpose Models - Foundation models for research and commercial fine-tuning applications.
- International Models - Foundational open-weights model series for research and commercial use.
- Large Language Models - Official implementation of the Llama model family.
- Open Source Models - Open and efficient foundation language models for research.
- General Purpose Models - Base model for numerous domain-specific fine-tuning projects.
- Large Language Models (LLMs) - Listed in the “Large Language Models (LLMs)” section of the The Incredible Pytorch awesome list.
