# datawhalechina/self-llm

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/datawhalechina-self-llm).**

28,285 stars · 2,822 forks · Jupyter Notebook · apache-2.0

## Links

- GitHub: https://github.com/datawhalechina/self-llm
- awesome-repositories: https://awesome-repositories.com/repository/datawhalechina-self-llm.md

## Topics

`chatglm` `chatglm3` `gemma-2b-it` `glm-4` `internlm2` `llama3` `llm` `lora` `minicpm` `q-wen` `qwen` `qwen1-5` `qwen2`

## Description

This project is an open-source educational resource providing structured, step-by-step guides for fine-tuning large language models. It focuses on adapting pre-trained transformer-based causal models to custom datasets, enabling users to transfer specific writing styles or domain knowledge into generative AI models.

The repository distinguishes itself by emphasizing parameter-efficient training techniques, specifically low-rank adaptation. By providing practical implementations for updating only a small subset of model weights, it allows for the customization of massive neural networks on consumer-grade hardware. The guides cover the entire machine learning workflow, including instruction-based dataset formatting, configuration of training parameters, and the use of gradient accumulation to manage memory constraints.

The documentation provides a comprehensive technical walkthrough for the fine-tuning process, from environment setup and data preparation to model training and weight saving. It includes specific code examples for loading models in half-precision formats and configuring training arguments to optimize performance for various tasks.

## Tags

### Artificial Intelligence & ML

- [Adapter Fine-Tuning](https://awesome-repositories.com/f/artificial-intelligence-ml/adapter-fine-tuning.md) — Injects trainable rank-decomposition matrices into transformer layers to update model behavior while keeping original weights frozen.
- [Fine-Tuning Frameworks](https://awesome-repositories.com/f/artificial-intelligence-ml/fine-tuning-frameworks.md) — Adapting pre-trained artificial intelligence models to specific datasets or tasks using efficient parameter-tuning techniques like LoRA.
- [Transformer Architectures](https://awesome-repositories.com/f/artificial-intelligence-ml/transformer-architectures.md) — Utilizes autoregressive architectures to predict subsequent tokens based on preceding context within a sequence.
- [Parameter Efficient Fine-Tuning](https://awesome-repositories.com/f/artificial-intelligence-ml/parameter-efficient-fine-tuning.md) — Optimizes memory usage by updating only a small subset of model parameters during the fine-tuning process.
- [Parameter-Efficient Tuning Techniques](https://awesome-repositories.com/f/artificial-intelligence-ml/parameter-efficient-tuning-techniques.md) — Optimizing the performance of massive neural networks on consumer-grade hardware by training only a small subset of model weights.
- [Fine-Tuning Tutorials](https://awesome-repositories.com/f/artificial-intelligence-ml/fine-tuning-tutorials.md) — 训练好了之后可以使用如下方式加载`lora`权重进行推理： ```python from transformers import AutoModelForCausalLM, AutoTokenizer import torch from peft import PeftModel mode_path = '/root/autodl-tmp/LLM-Research/Meta-Llama-3-8B-Instruct' lora_path ([source](https://github.com/datawhalechina/self-llm/blob/master/models/LLaMA3/04-LLaMA3-8B-Instruct%20Lora%20%E5%BE%AE%E8%B0%83.md))
- [Knowledge Distillation Tools](https://awesome-repositories.com/f/artificial-intelligence-ml/knowledge-distillation-tools.md) — Teaching a general-purpose language model to adopt a specific writing style or domain knowledge from a provided text corpus.
- [Training Optimizations](https://awesome-repositories.com/f/artificial-intelligence-ml/training-optimizations.md) — Simulates larger batch sizes by aggregating gradients over multiple forward and backward passes before updating model weights.
- [Training Frameworks](https://awesome-repositories.com/f/artificial-intelligence-ml/training-frameworks.md) — ```python trainer = Trainer( model=model, args=args, train_dataset=tokenized_id, data_collator=DataCollatorForSeq2Seq(tokenizer=tokenizer, padding=True), ) trainer.train() ``` ([source](https://github.com/datawhalechina/self-llm/blob/master/models/LLaMA3/04-LLaMA3-8B-Instruct%20Lora%20%E5%BE%AE%E8%B0%83.md))
- [Precision Quantization](https://awesome-repositories.com/f/artificial-intelligence-ml/precision-quantization.md) — Reduces memory footprint and accelerates computation by representing model weights in lower-bit floating point formats.
- [Instruction Tuning Datasets](https://awesome-repositories.com/f/artificial-intelligence-ml/instruction-tuning-datasets.md) — Structures raw text data into prompt-response pairs to align large language models with specific task requirements.

### Education & Learning Resources

- [Artificial Intelligence Curricula](https://awesome-repositories.com/f/education-learning-resources/artificial-intelligence-curricula.md) — Provides practical instructions and community-driven knowledge for deploying and customizing machine learning models.
- [Fine-Tuning Tutorials](https://awesome-repositories.com/f/education-learning-resources/fine-tuning-tutorials.md) — Provides instructional guidance on configuring LoRA parameters for fine-tuning large language models. ([source](https://github.com/datawhalechina/self-llm/blob/master/models/LLaMA3/04-LLaMA3-8B-Instruct%20Lora%20%E5%BE%AE%E8%B0%83.md))
- [Model Optimization Guides](https://awesome-repositories.com/f/education-learning-resources/model-optimization-guides.md) — Offers technical guidance on implementing low-rank adaptation techniques to optimize model performance with minimal overhead.
- [Fine-Tuning Guides](https://awesome-repositories.com/f/education-learning-resources/fine-tuning-guides.md) — 本文基础环境如下： ``` ---------------- ubuntu 22.04 python 3.12 cuda 12.1 pytorch 2.3.0 ---------------- ``` > 本文默认学习者已安装好以上 Pytorch(cuda) 环境，如未安装请自行安装。 首先 `pip` 换源加速下载并安装依赖包 ```shell # 升级pip python -m pip install --upgrade pip ([source](https://cdn.jsdelivr.net/gh/datawhalechina/self-llm@master/examples/Chat-嬛嬛/readme.md#chat))
- [Model Training Guides](https://awesome-repositories.com/f/education-learning-resources/model-training-guides.md) — A collection of step-by-step guides and code examples for training and adapting large language models on custom datasets.

### Software Engineering & Architecture

- [Training Pipelines](https://awesome-repositories.com/f/software-engineering-architecture/training-pipelines.md) — A structured approach to preparing data, configuring training parameters, and managing model checkpoints for generative language tasks.