Awesome GitHub RepositoriesChinese NLP Libraries

HanLP is a natural language processing library and deep learning framework specifically optimized for the Chinese language, while also functioning as a multilingual text processor. It serves as a toolkit for performing linguistic analysis, semantic understanding, and script conversion. The project distinguishes itself through a dedicated focus on Chinese linguistic structures, including a specialized script converter for transforming text between Simplified Chinese, Traditional Chinese, and Pinyin. It further supports domain-specific model training to improve the recognition of professional t

This project is a collection of pre-trained dense and sparse word vectors trained on diverse Chinese corpora. It serves as a library of linguistic representations and an NLP vector dataset designed to improve the accuracy of semantic and morphological analysis in text models. The collection provides corpus-specific representations and utilizes n-gram co-occurrence modeling to capture diverse linguistic patterns. It includes a hybrid of dense-sparse vectors to balance computational efficiency and semantic precision. The project covers semantic vector search and the development of Chinese natu

Chinese-BERT-wwm is a pre-trained transformer model and encoder designed for Chinese natural language processing. It converts Chinese text into dense vector representations to be used across various natural language processing applications. The model utilizes a whole word masking strategy during pre-training, masking entire words rather than individual characters. This approach is designed to improve the capture of semantic meaning and language structure within Chinese datasets. The project covers a range of downstream tasks including text classification, sequence labeling, and reading compr

pkuseg-python is a Chinese word segmentation toolkit and natural language processing library. It provides specialized models for splitting Chinese text into words across various domains, including news, medical, and web content, and includes a tool for assigning grammatical parts of speech tags to segmented words. The library allows for the training of custom segmentation models using annotated datasets and supports the integration of user-defined dictionaries to ensure specialized terminology is recognized correctly. It employs a multi-threaded execution engine to process large volumes of Ch

SnowNLP is a Python library for Chinese natural language processing. It provides tools for text segmentation, sentiment analysis, document classification, and phonetic transliteration. The library includes capabilities for training and saving custom machine learning models for tokenization and sentiment analysis using raw training datasets. It covers a range of linguistic processing areas, including parts of speech tagging, sentence splitting, and text similarity measurement. The toolkit also provides utilities for extracting key information through text summarization and calculating word im

ansj_seg is a Java NLP toolkit and segmentation library designed for processing Chinese text. It functions as a word segmenter, part-of-speech tagger, and named entity recognizer to divide continuous Chinese characters into meaningful words and tokens. The library utilizes statistical models for text segmentation and provides capabilities for identifying and extracting person names from unstructured documents. It also assigns grammatical categories to tokens to determine their linguistic roles within a sentence. The toolkit supports domain-specific text processing through the use of custom d

Ce projet fournit une collection curatée de mots chinois à haute fréquence et non informatifs, issus de standards académiques et industriels. Il sert de jeu de données de référence et de collection de mots vides (stopwords) conçue pour être utilisée dans des tâches de traitement du langage naturel (NLP). Le dépôt se concentre sur le prétraitement du texte chinois pour réduire le bruit et améliorer la précision des modèles de machine learning. Il fournit des jeux de données filtrés spécifiquement pour la recherche d'informations en chinois, la préparation à l'analyse de sentiment et le nettoyage général des données. Le projet utilise des lexiques pré-compilés et un stockage en fichiers plats pour permettre un filtrage efficace des mots vides et une agrégation de vocabulaire pour les corpus chinois.

Il s'agit d'une boîte à outils de traitement du langage naturel chinois fournissant une suite d'outils pour la segmentation des mots, l'étiquetage morphosyntaxique et la reconnaissance d'entités nommées. Elle inclut un analyseur de dépendances neuronal pour analyser les relations syntaxiques et sémantiques entre les mots, ainsi qu'une suite d'entraînement en apprentissage automatique pour créer des modèles linguistiques personnalisés en utilisant des jeux de données annotés. La boîte à outils se distingue par sa flexibilité de déploiement, offrant un serveur dockerisé et une interface de service web qui expose les capacités de traitement via API. Elle prend en charge l'utilisation de modèles pré-entraînés et permet l'intégration de lexiques externes et d'extensions de dictionnaires de mots pour améliorer la précision de l'analyse. Globalement, le projet couvre un pipeline complet de tâches linguistiques, incluant la segmentation des phrases, le mappage des dépendances syntaxiques et l'étiquetage des rôles sémantiques. Ces capacités sont disponibles via une interface en ligne de commande, des modules autonomes ou des pipelines d'analyse intégrés. La logique centrale est implémentée en C++ avec des bindings officiels pour Python et Java.

LAC is a Chinese lexical analysis engine and toolkit designed for joint word segmentation, part-of-speech tagging, and named entity recognition. It functions as a high-performance system that identifies word boundaries and grammatical categories using trained machine learning models. The project features a lightweight, compiled native runtime that enables on-device natural language processing and embedding into mobile applications. It includes model compression and conversion to optimize for resource-constrained environments and supports multi-threaded parallel execution to increase throughpu