This project is an educational implementation of a small-scale generative pre-trained transformer designed to teach the fundamentals of neural network architecture and training. It serves as a reference implementation and tutorial for constructing a text-generating neural network from scratch. The codebase demonstrates the mechanics of tokenization, self-attention, and the construction of a lightweight language model. It focuses on the step-by-step process of building a generative model to illustrate how large language models are constructed. The implementation covers transformer-based archi
中文对话0.2B小模型（ChatLM-Chinese-0.2B），开源所有数据集来源、数据清洗、tokenizer训练、模型预训练、SFT指令微调、RLHF优化等流程的全部代码。支持下游任务sft微调，给出三元组信息抽取微调示例。
The main features of openbmb/forgetrain are: Small Language Models.
Open-source alternatives to openbmb/forgetrain include: karpathy/ng-video-lecture — This project is an educational implementation of a small-scale generative pre-trained transformer designed to teach… charent/chatlm-mini-chinese — 中文对话0.2B小模型（ChatLM-Chinese-0.2B），开源所有数据集来源、数据清洗、tokenizer训练、模型预训练、SFT指令微调、RLHF优化等流程的全部代码。支持下游任务sft微调，给出三元组信息抽取微调示例。. charent/phi2-mini-chinese. chinese-tiny-llm/chinese-tiny-llm. dllxw/baby-llama2-chinese — This project is a training pipeline and framework for developing Chinese language models based on the Llama 2… allenai/olmo.
