# yuanzhoulvpi2017/zero_nlp

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/yuanzhoulvpi2017-zero-nlp).**

_How this analysis was created: the description and tags below were written by an AI model that read this project's README and public documentation pages; stars, license and language come straight from the GitHub API. The model does not read the source code._

3,825 stars · 444 forks · Jupyter Notebook · MIT

## Links

- GitHub: https://github.com/yuanzhoulvpi2017/zero_nlp
- awesome-repositories: https://awesome-repositories.com/repository/yuanzhoulvpi2017-zero-nlp.md

## Topics

`bert` `chatglm-6b` `clip` `gpt` `gpt2` `huggingface-transformers` `llama` `llama2` `llava` `nlp` `pytorch` `text-generation` `transformers`

## Description

zero_nlp is a distributed framework for training and fine-tuning large language models and multimodal architectures. It provides a specialized toolkit for distributed model parallelism, allowing neural network layers and weights to be partitioned across multiple GPU devices to train models that exceed the memory capacity of a single processor.

The project distinguishes itself through a combination of high-throughput data pipelines and parameter-efficient tuning. It utilizes multi-threading and memory mapping to preprocess and stream datasets exceeding 100GB and implements memory-saving adaptation techniques such as Low-Rank Adaptation and Direct Preference Optimization.

The system covers a broad range of capabilities, including multimodal model development for image captioning and feature extraction, vocabulary management for expanding or pruning tokenizers, and the deployment of models via web APIs and chatbot interfaces. It also supports training workflows for causal language models, text classifiers, and contrastive learning.

The project is implemented using Jupyter Notebooks.

## Tags

### Artificial Intelligence & ML

- [Distributed GPU Training](https://awesome-repositories.com/f/artificial-intelligence-ml/distributed-gpu-training.md) — Distributes computational workloads across multiple GPUs to accelerate training via model parallelism. ([source](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/simple_thu_chatglm6b))
- [Distributed Model Parallelism](https://awesome-repositories.com/f/artificial-intelligence-ml/distributed-model-parallelism.md) — Implements a specialized toolkit for partitioning neural network layers and weights across multiple GPUs for large-scale training.
- [Parameter Partitioning](https://awesome-repositories.com/f/artificial-intelligence-ml/large-scale-training/model-parameter-scaling/parameter-partitioning.md) — The project distributes model states and gradients across multiple devices to train large architectures on limited hardware. ([source](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/chinese_bloom))
- [Data-Parallel Training](https://awesome-repositories.com/f/artificial-intelligence-ml/distributed-training-frameworks/data-parallel-training.md) — Distributes model training across multiple processes by partitioning data and synchronizing gradients. ([source](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/pytorch_base))
- [Generative Model Training Tools](https://awesome-repositories.com/f/artificial-intelligence-ml/generative-model-training-tools.md) — Trains generative models on custom text datasets to produce specific content styles. ([source](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/chinese_gpt2))
- [Language Model Fine-Tuning](https://awesome-repositories.com/f/artificial-intelligence-ml/language-model-fine-tuning.md) — Adapts pre-trained models to specific behaviors and knowledge using supervised fine-tuning on task-specific datasets. ([source](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/train_qwen))
- [Large Scale Dataset Processing](https://awesome-repositories.com/f/artificial-intelligence-ml/large-scale-dataset-processing.md) — Features high-throughput pipelines for parallel loading and preprocessing of hundred-gigabyte datasets using multi-threading and memory mapping.
- [Large-Scale Model Training](https://awesome-repositories.com/f/artificial-intelligence-ml/large-scale-model-training.md) — Distributes model weights and computations across hardware to enable training of massive architectures. ([source](https://github.com/yuanzhoulvpi2017/zero_nlp/blob/main/requirements.txt))
- [Layer Partitioning](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/frameworks/model-construction/neural-network-layers/multi-layer-architectures/layer-partitioning.md) — Implements layer-based model partitioning to train architectures that exceed the memory of a single GPU.
- [Low-Rank Adaptation](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/infrastructure/model-training-and-tuning/fine-tuning-and-customization/model-fine-tuning/low-rank-adaptation.md) — Implements Low-Rank Adaptation (LoRA) to update a small subset of parameters, reducing memory overhead.
- [Large Language Model Training Frameworks](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/infrastructure/model-training-and-tuning/training-frameworks/large-language-model-training-frameworks.md) — Provides a distributed framework for training and fine-tuning large transformer-based models across multi-GPU environments.
- [Language Model Training](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/model-fine-tuning-adaptation/language-model-training.md) — Implements training and fine-tuning workflows for various large language model architectures. ([source](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/model_clm))
- [Multi-GPU Distribution](https://awesome-repositories.com/f/artificial-intelligence-ml/model-optimization/inference-deployment/model-deployment-toolkits/distributed-deployment-utilities/multi-gpu-distribution.md) — Distributes model weights across multiple GPUs via an orchestration layer for large-scale inference and training. ([source](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/qwen3_embedding))
- [Weight Distribution](https://awesome-repositories.com/f/artificial-intelligence-ml/model-weight-management/weight-distribution.md) — Distributes model workloads and parameters across multiple hardware devices using communication strategies. ([source](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/chinese_clip_ddp))
- [Multimodal Model Integrations](https://awesome-repositories.com/f/artificial-intelligence-ml/multimodal-model-integrations.md) — Combines vision and language models into a single architecture to process visual inputs and generate text. ([source](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/train_llava))
- [Pipeline Parallelisms](https://awesome-repositories.com/f/artificial-intelligence-ml/neural-networks/model-training-pipelines/pipeline-parallelisms.md) — Distributes model weights and computation across a sequence of GPUs using pipeline parallelism.
- [Parameter Efficient Fine-Tuning](https://awesome-repositories.com/f/artificial-intelligence-ml/parameter-efficient-fine-tuning.md) — Implements memory-efficient adaptation techniques like Low-Rank Adaptation (LoRA) to reduce computational overhead during fine-tuning.
- [Pipeline Parallelism Implementations](https://awesome-repositories.com/f/artificial-intelligence-ml/pipeline-parallelism-implementations.md) — Implements techniques for partitioning model layers across multiple devices to train architectures exceeding single-GPU memory. ([source](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/pipeline))
- [Pipeline Parallelism Partitioners](https://awesome-repositories.com/f/artificial-intelligence-ml/pipeline-parallelism-partitioners.md) — Provides utilities to partition large neural networks into sequential layers across multiple GPUs for pipeline-parallel training. ([source](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/pipeline))
- [Flash-Attention Implementations](https://awesome-repositories.com/f/artificial-intelligence-ml/attention-mechanisms/flash-attention-implementations.md) — Utilizes flash attention implementations to optimize memory usage and processing speed for model training and inference. ([source](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/model_baichuan))
- [Direct Preference Optimization](https://awesome-repositories.com/f/artificial-intelligence-ml/custom-model-training/reward-modeling/preference-alignment/direct-preference-optimization.md) — Optimizes language models using Direct Preference Optimization to improve response quality. ([source](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/baichuan2_dpo))
- [Domain-Specific Data Preparation](https://awesome-repositories.com/f/artificial-intelligence-ml/domain-specific-data-preparation.md) — Cleans and transforms raw text using templates to format vertical-domain datasets for model consumption. ([source](https://github.com/yuanzhoulvpi2017/zero_nlp/blob/main/README.md))
- [Incremental Vocabulary Adaptation](https://awesome-repositories.com/f/artificial-intelligence-ml/embedding-adaptation-utilities/vocabulary-embedding-adapters/incremental-vocabulary-adaptation.md) — Supports expanding tokenizer embedding layers and adapting weights to incorporate domain-specific terminology.
- [End-to-End Training Pipelines](https://awesome-repositories.com/f/artificial-intelligence-ml/end-to-end-training-pipelines.md) — Executes integrated workflows for training, fine-tuning, and deploying generative and multimodal architectures. ([source](https://github.com/yuanzhoulvpi2017/zero_nlp#readme))
- [Multimodal Feature Encoding](https://awesome-repositories.com/f/artificial-intelligence-ml/feature-extraction/multimodal-feature-encoding.md) — Develops multimodal systems that convert visual data into textual captions or embeddings for language model consumption.
- [Multimodal Feature Extractors](https://awesome-repositories.com/f/artificial-intelligence-ml/feature-extraction/multimodal-feature-extractors.md) — Extracts unified vector representations from text and images using separate encoders for matching or classification. ([source](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/chinese_clip))
- [Gradient Checkpointing](https://awesome-repositories.com/f/artificial-intelligence-ml/gradient-checkpointing.md) — Employs gradient checkpointing to reduce peak memory requirements when training large models on limited hardware. ([source](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/simple_thu_chatglm6b))
- [Caption Generators](https://awesome-repositories.com/f/artificial-intelligence-ml/image-captioning-training/caption-generators.md) — Combines vision encoders and language decoders to generate natural language descriptions for visual content. ([source](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/vit-gpt2-image-chinese-captioning))
- [Contrastive Pre-training](https://awesome-repositories.com/f/artificial-intelligence-ml/large-scale-model-training/vision-transformer-pre-training/contrastive-pre-training.md) — Trains models to learn matching relationships between text and image pairs using contrastive learning. ([source](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/chinese_clip))
- [Data Preparation Tools](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/infrastructure/data-ingestion-preparation/data-preparation-tools.md) — Cleans and converts raw text from JSON or CSV files into formats suitable for supervised fine-tuning. ([source](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/simple_thu_chatglm6b))
- [Mixed Precision Training](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/infrastructure/machine-learning-training/distributed-and-accelerated-compute/training-acceleration-tools/mixed-precision-training.md) — Uses bfloat16 mixed precision training to reduce memory overhead and improve stability during the training process. ([source](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/chinese_dolly_v2_3b))
- [Fully Sharded Data Parallelism](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/infrastructure/model-training-and-tuning/distributed-and-scaling-strategies/distributed-learning/fully-sharded-data-parallelism.md) — Computes output probability distributions using fully sharded data parallelism to manage large-scale weights. ([source](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/verl_base))
- [Multimodal Model Pipelines](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/infrastructure/model-training-and-tuning/training-frameworks/model-training-pipelines/multimodal-model-pipelines.md) — Implements training pipelines that integrate vision encoders and language decoders for multimodal image and text tasks.
- [Text Generation Pipelines](https://awesome-repositories.com/f/artificial-intelligence-ml/text-generation-pipelines.md) — Implements an inference pipeline that loads model checkpoints to process prompts and generate text outputs. ([source](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/chinese_gpt2))
- [Vocabulary Expansion](https://awesome-repositories.com/f/artificial-intelligence-ml/text-tokenizers/vocabulary-expansion.md) — Provides utilities to expand tokenizer embedding layers to incorporate new domain-specific tokens.

### Data & Databases

- [Streaming Preprocessing Pipelines](https://awesome-repositories.com/f/data-databases/data-preprocessing-pipelines/streaming-preprocessing-pipelines.md) — Ships a high-throughput data system that streams and transforms massive datasets incrementally to prevent system memory exhaustion.
- [Large-Scale Data Preprocessing](https://awesome-repositories.com/f/data-databases/large-scale-data-preprocessing.md) — The project handles hundred-gigabyte scale data using multi-threading and memory mapping for efficient loading and preprocessing. ([source](https://github.com/yuanzhoulvpi2017/zero_nlp/blob/main/README.md))
- [Multi-GPU Layer Distribution](https://awesome-repositories.com/f/data-databases/memory-optimization-strategies/compositor-memory-limits/model-layer-offloading/multi-gpu-layer-distribution.md) — Splits model layers across multiple GPUs to enable the training of models that exceed single-device memory. ([source](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/chatglm_v2_6b_lora))
- [Large Dataset Streaming](https://awesome-repositories.com/f/data-databases/incremental-data-streaming/large-dataset-streaming.md) — Processes training data incrementally from disk to handle datasets exceeding 100GB without memory exhaustion. ([source](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/model_clm))
- [Memory-Mapped File Access](https://awesome-repositories.com/f/data-databases/large-scale-dataset-management/memory-mapped-file-access.md) — Utilizes memory-mapped file access to read massive datasets incrementally from disk without exhausting system memory.

### Software Engineering & Architecture

- [Distributed Training Coordination](https://awesome-repositories.com/f/software-engineering-architecture/distributed-coordination-systems/distributed-training-coordination.md) — Coordinates computation tasks across multiple hardware devices using a distributed execution framework. ([source](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/verl_base))
- [Multi-Threaded Data Pipelines](https://awesome-repositories.com/f/software-engineering-architecture/multi-threaded-batch-processing/multi-threaded-data-pipelines.md) — Employs multi-threaded data pipelines to transform raw text and images for higher training throughput.

### Part of an Awesome List

- [Cross-Modal Models](https://awesome-repositories.com/f/awesome-lists/ai/cross-modal-models.md) — Aligns output dimensions of disparate model encoders using linear layers to enable joint feature matching. ([source](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/chinese_clip))
- [Natural Language Processing](https://awesome-repositories.com/f/awesome-lists/ai/natural-language-processing.md) — Listed in the “Natural Language Processing” section of the FunNLP awesome list.