awesome-repositories.com
© 2026 Bringes Technology SRL·VAT RO45896025·hello@bringes.io
MCPSitemapPrivacyTerms
Self Llm | Awesome Repository
← All repositories

datawhalechina/self-llm

0
View on GitHub↗
28,285 stars·2,822 forks·Jupyter Notebook·apache-2.0·0 views

Self Llm

Features

  • Adapter Fine-Tuning - Injects trainable rank-decomposition matrices into transformer layers to update model behavior while keeping original weights frozen.
  • Fine-Tuning Frameworks - Adapting pre-trained artificial intelligence models to specific datasets or tasks using efficient parameter-tuning techniques like LoRA.
  • Transformer Architectures - Utilizes autoregressive architectures to predict subsequent tokens based on preceding context within a sequence.
  • Artificial Intelligence Curricula - Provides practical instructions and community-driven knowledge for deploying and customizing machine learning models.
  • Parameter Efficient Fine-Tuning - Optimizes memory usage by updating only a small subset of model parameters during the fine-tuning process.
  • Parameter-Efficient Tuning Techniques - Optimizing the performance of massive neural networks on consumer-grade hardware by training only a small subset of model weights.
  • Fine-Tuning Tutorials - 训练好了之后可以使用如下方式加载lora权重进行推理： ```python from transformers import AutoModelForCausalLM, AutoTokenizer import torch from peft import PeftModel modepath = '/root/autodl-tmp/LLM-Research/Meta-Llama-3-8B-Instruct' lorapath
  • Knowledge Distillation Tools - Teaching a general-purpose language model to adopt a specific writing style or domain knowledge from a provided text corpus.
  • Training Optimizations - Simulates larger batch sizes by aggregating gradients over multiple forward and backward passes before updating model weights.
  • Fine-Tuning Tutorials - Provides instructional guidance on configuring LoRA parameters for fine-tuning large language models.
  • Model Optimization Guides - Offers technical guidance on implementing low-rank adaptation techniques to optimize model performance with minimal overhead.
  • Training Frameworks - Comprehensive toolkits for configuring, executing, and benchmarking model training pipelines. Distinguishing note: Focuses on the full training lifecycle including benchmarking, rather than just the training algorithm.
  • Precision Quantization - Reduces memory footprint and accelerates computation by representing model weights in lower-bit floating point formats.
  • Training Pipelines - A structured approach to preparing data, configuring training parameters, and managing model checkpoints for generative language tasks.
  • Instruction Tuning Datasets - Structures raw text data into prompt-response pairs to align large language models with specific task requirements.
  • Fine-Tuning Guides - 本文基础环境如下： > 本文默认学习者已安装好以上 Pytorch(cuda) 环境，如未安装请自行安装。 首先 pip 换源加速下载并安装依赖包 ```shell # 升级pip python -m pip install --upgrade pip
  • Model Training Guides - A collection of step-by-step guides and code examples for training and adapting large language models on custom datasets.

    • This project is an open-source educational resource providing structured, step-by-step guides for fine-tuning large language models. It focuses on adapting pre-trained transformer-based causal models to custom datasets, enabling users to transfer specific writing styles or domain knowledge into generative AI models.

    The repository distinguishes itself by emphasizing parameter-efficient training techniques, specifically low-rank adaptation. By providing practical implementations for updating only a small subset of model weights, it allows for the customization of massive neural networks on consumer-grade hardware. The guides cover the entire machine learning workflow, including instruction-based dataset formatting, configuration of training parameters, and the use of gradient accumulation to manage memory constraints.

    The documentation provides a comprehensive technical walkthrough for the fine-tuning process, from environment setup and data preparation to model training and weight saving. It includes specific code examples for loading models in half-precision formats and configuring training arguments to optimize performance for various tasks.