codemayq/chinese-chatbot-corpus
Chinese Chatbot Corpus
Dieses Projekt bietet eine Sammlung verarbeiteter chinesischer Konversationsdatensätze und Preprocessing-Workflows, die für das Training und Instruction-Tuning von Large Language Models (LLMs) entwickelt wurden. Es fungiert als Trainingskorpus aus bereinigtem, standardisiertem chinesischem Text, der als Frage-Antwort-Paare formatiert ist.
Das Repository enthält eine Preprocessing-Pipeline und einen Datensatz-Aggregator, die mehrere öffentliche Chat-Quellen in einheitliche Dateien zusammenführen. Diese Tools normalisieren den Text, indem sie traditionelle chinesische Schriftzeichen in vereinfachte Zeichen umwandeln und komplexe Dialogstränge in eine standardisierte Sequenz von Einzel-Turns transformieren.
Die resultierenden Daten werden als unabhängige, tabulatorgetrennte Dateien exportiert, was ein konsistentes Schema über verschiedene Quellen hinweg sicherstellt. Dieser technische Workflow konzentriert sich auf die Beseitigung von Formatierungsinkonsistenzen, um rohe Konversationsdaten für Machine-Learning-Aufgaben vorzubereiten.
Features
- Training Datasets - Provides large sets of processed conversational Chinese text for training and fine-tuning LLMs.
- Dialogue Turn Flatteners - Transforms complex dialogue threads into standardized query-answer pairs suitable for model instruction tuning.
- Chinese QA Datasets - Provides a cleaned collection of Chinese question-and-answer pairs for model training.
- Conversational Dataset Sourcing - Structures and acquires dialogue data from multiple public sources for training generative AI models.
- Corpus Compilation - Compiles and structures raw conversational text into a unified corpus for language model training.
- Chinese Script Normalizers - Standardizes Chinese text by converting script variants and formatting raw dialogue into uniform pairs.
- Chinese Script Normalization Pipelines - Implements a preprocessing pipeline that normalizes Chinese script variants and standardizes dialogue turns.
- Conversational Dataset Aggregators - Combines multiple public chat sources into unified files while removing formatting inconsistencies.
- Corpus Aggregators - Aggregates multiple public chat sources into a single consolidated repository with a shared schema.
- Data Standardization - Normalizes disparate chat formats and script variants into consistent, standardized dialogue structures.
- Chinese Character Simplifiers - Provides utilities to convert traditional Chinese characters to simplified script for data consistency.
- Instruction Tuning Datasets - Provides processed TSV files formatted to align language models with conversational instructions in Chinese.
- Training Dataset Processing - Processes large-scale conversational datasets into structured formats specifically for machine learning training.
- Data Cleaning Procedures - Cleans raw conversational data to ensure every row represents a valid query-answer pair.
- Processing Pipelines - Implements data processing pipelines to handle raw conversational datasets before merging them into a unified structure.
- File-Based Data Exports - Serializes processed conversational pairs into portable tab-separated value files.
- Natural Language Processing - Listed in the “Natural Language Processing” section of the FunNLP awesome list.
