codemayq/chinese-chatbot-corpus
Chinese Chatbot Corpus
This project provides a collection of processed Chinese conversational datasets and preprocessing workflows designed for training and instruction tuning of large language models. It functions as a training corpus of cleaned, standardized Chinese text formatted as query-answer pairs.
The repository includes a preprocessing pipeline and dataset aggregator that combine multiple public chat sources into unified files. These tools normalize text by converting traditional Chinese characters to simplified characters and transforming complex dialogue threads into a standardized sequence of single turns.
The resulting data is exported as independent tab-separated value files, ensuring a consistent schema across disparate sources. This engineering workflow focuses on removing formatting inconsistencies to prepare raw conversational data for machine learning tasks.
Features
- Training Datasets - Provides large sets of processed conversational Chinese text for training and fine-tuning LLMs.
- Dialogue Turn Flatteners - Transforms complex dialogue threads into standardized query-answer pairs suitable for model instruction tuning.
- Chinese QA Datasets - Provides a cleaned collection of Chinese question-and-answer pairs for model training.
- Conversational Dataset Sourcing - Structures and acquires dialogue data from multiple public sources for training generative AI models.
- Corpus Compilation - Compiles and structures raw conversational text into a unified corpus for language model training.
- Chinese Script Normalizers - Standardizes Chinese text by converting script variants and formatting raw dialogue into uniform pairs.
- Chinese Script Normalization Pipelines - Implements a preprocessing pipeline that normalizes Chinese script variants and standardizes dialogue turns.
- Conversational Dataset Aggregators - Combines multiple public chat sources into unified files while removing formatting inconsistencies.
- Corpus Aggregators - Aggregates multiple public chat sources into a single consolidated repository with a shared schema.
- Data Standardization - Normalizes disparate chat formats and script variants into consistent, standardized dialogue structures.
- Chinese Character Simplifiers - Provides utilities to convert traditional Chinese characters to simplified script for data consistency.
- Instruction Tuning Datasets - Provides processed TSV files formatted to align language models with conversational instructions in Chinese.
- Training Dataset Processing - Processes large-scale conversational datasets into structured formats specifically for machine learning training.
- Data Cleaning Procedures - Cleans raw conversational data to ensure every row represents a valid query-answer pair.
- Processing Pipelines - Implements data processing pipelines to handle raw conversational datasets before merging them into a unified structure.
- File-Based Data Exports - Serializes processed conversational pairs into portable tab-separated value files.
- Natural Language Processing - Listed in the “Natural Language Processing” section of the FunNLP awesome list.
