TinyLlama

Features

Generative AI Models - Provides a lightweight 1.1B parameter generative model optimized for deployment on edge hardware.
Data-Parallel Training - Provides a framework for distributing training workloads across multiple GPUs using data parallelism.
Efficient LLM Development - Focuses on the development and optimization of compact models to enable high performance on constrained hardware.
LLM Development Toolkits - Includes a toolkit for full-parameter weight adjustments to adapt the model for specific tasks.
Large Language Model Training Frameworks - Executes distributed pretraining of transformer models across multi-GPU environments using optimized attention.
Model Pretraining Frameworks - Provides a distributed system for training foundational language models across multiple GPUs and nodes.
Finetuning Workflows - Provides workflows for adapting a pretrained base model to specific tasks or chat styles.
Language Model Pretraining - Trains a compact Llama model on a 3 trillion token dataset using distributed GPU clusters.
On-Device Text Generation Runners - Enables real-time text generation and dialogue execution on memory-constrained edge hardware.
Language Models - Implements a compact 1.1B parameter language model for high-performance text generation.
FlashAttention - Uses FlashAttention kernels to minimize GPU memory access and increase attention computation speed.
Full Parameter Fine-Tuning - Ships scripts for updating all model parameters to adapt the general model for chat interactions.
Causal Masking - Implements causal masking to prevent the model from attending to future tokens during training.
RMSNorm Layers - Implements root mean square layer normalization to stabilize neural network activations during training.
Rotary Positional Embeddings - Utilizes rotary positional embeddings to encode relative token positions in a high-dimensional space.
Pretrained Weight Initializers - Employs specific weight initialization distributions to ensure stable convergence during large-scale pretraining.
Large Language Models - Compact language models for efficient deployment.
Small Language Models - Pre-trained small-scale LLaMA model.
Large Language Models (LLMs) - Listed in the “Large Language Models (LLMs)” section of the The Incredible Pytorch awesome list.

Open-source alternatives to TinyLlama

Similar open-source projects, ranked by how many features they share with TinyLlama.

lightning-ai/litgpt
Lightning-AI/litgpt
13,431View on GitHub
LitGPT is a training and deployment framework for large language models, providing a suite of tools for pretraining, finetuning, quantizing, evaluating, and serving models within a production environment. It includes a dedicated training pipeline for adapting pretrained models to specific tasks, a quantization tool for reducing weight precision, and an inference server for hosting models via web interfaces. The framework supports high-performance model development through custom architecture implementation and the use of predefined recipes to standardize pretraining and finetuning. It enables
Python
View on GitHub13,431
karpathy/llm.c
karpathy/llm.c
30,230View on GitHub
This project is a low-dependency engine designed for training large language models using native C and CUDA. It provides a bare-metal environment for tensor computation, allowing for the execution of neural network operations directly on hardware accelerators without the overhead of high-level software abstractions. The framework distinguishes itself by implementing manual gradient backpropagation and custom hardware-specific kernels, providing granular control over memory mapping and computational precision. It supports distributed training across multiple graphics processors and compute nod
Cuda
View on GitHub30,230
deepseek-ai/deepseek-llm
deepseek-ai/deepseek-LLM
7,100View on GitHub
DeepSeek-LLM is a large language model and causal language model designed for natural language generation. It functions as a multi-lingual system capable of predicting the next token in a sequence to perform text completion and conversational generation. The model is specialized for logical reasoning, specifically as a code and math LLM. This enables it to perform complex problem solving, which includes generating executable code and solving mathematical equations through step-by-step analysis. The system's broader capabilities cover conversational AI, including the generation of chat comple
Makefile
View on GitHub7,100
d2l-ai/d2l-en
d2l-ai/d2l-en
29,001View on GitHub
This project is an educational platform and research toolkit designed to teach deep learning through a combination of mathematical theory, visual diagrams, and executable code. It provides a comprehensive environment for building, training, and evaluating neural networks, grounding complex concepts in interactive computational notebooks that allow for hands-on experimentation. The framework distinguishes itself by interleaving theoretical foundations—including linear algebra, calculus, and probability—with practical implementations across multiple industry-standard libraries. It supports flex
Pythonbookcomputer-visiondata-science
View on GitHub29,001

See all 30 alternatives to TinyLlama

jzhang38TinyLlamaArchived

Features

Open-source alternatives to TinyLlama

Lightning-AI/litgpt

karpathy/llm.c

deepseek-ai/deepseek-LLM

d2l-ai/d2l-en

Star history

Open-source alternatives to TinyLlama

Lightning-AI/litgpt

karpathy/llm.c

deepseek-ai/deepseek-LLM

d2l-ai/d2l-en