dongrixinyu/JioNLP
JioNLP
JioNLP is a Chinese natural language processing toolkit designed for cleaning, normalizing, and extracting structured information from unstructured text. It functions as a linguistic analyzer for Chinese characters and a rule-based named entity extractor, providing a specialized system for sentiment scoring and synthetic data generation for machine learning workflows.
The project features a lexicon-based sentiment analysis engine that computes numerical emotional tone scores and a data augmentation library that uses back-translation and synonym replacement to expand training datasets. It includes a rule-based entity extraction system that parses dates, currency, addresses, and identity numbers into standardized formats.
The toolkit covers broad capability areas including information extraction for geographic and temporal data, script conversion between simplified and traditional Chinese, and character-level mapping for pinyin and stroke counts. It also provides utilities for corpus management, such as stopword filtering, key phrase extraction, and the removal of noise like HTML and URLs from raw text.
Features
- Chinese Natural Language Processing - Provides a comprehensive suite for cleaning, normalizing, and analyzing Chinese text for machine learning.
- Pinyin Character Mappings - Maps Chinese characters to their structural components, stroke counts, and pinyin representations.
- Entity and Relation Extraction - Identifies entities and specific terms by matching input text against predefined linguistic dictionaries.
- Structured Entity Extractors - Provides automated extraction and normalization of structured data points like dates, currency, and ID numbers from raw text.
- Chinese Entity Recognizers - Identifies and normalizes structured Chinese data including dates, currency, and geographic locations.
- Geographic Entity Extraction - Identifies geographic entities and parses address strings into administrative levels for domestic and international locations.
- Chinese Character Structural Analyzers - Performs stroke counting, radical identification, and pinyin generation for Chinese characters.
- Named Entity Recognition - Identifies and classifies specific entities within unstructured text using a dictionary-based matching approach.
- Script Conversion - Transforms text between simplified and traditional Chinese characters using standardized mapping dictionaries.
- Lexicon-Based Sentiment Analyzers - Calculates emotional polarity by summing pre-assigned weights from a weighted lexicon.
- Chinese Sentiment Analyzers - Quantifies emotional tone in Chinese text using specialized weighted lexicons.
- Script Converters - Transforms text between simplified and traditional Chinese scripts and generates phonetic pinyin representations.
- Chinese NLP Toolkits - A comprehensive library for cleaning, normalizing, and extracting structured information from unstructured Chinese text.
- Text Cleaning Utilities - Cleans raw text by removing HTML tags, URLs, and noise while normalizing character widths.
- Pattern-Based Extraction - Uses regular expressions and structured masks to parse ID numbers, dates, and currency.
- Text Input Normalizers - Normalizes unstructured text by removing HTML, URLs, and inconsistent whitespace.
- Automated Text Analysis - Implements word frequency analysis and probabilistic modeling to identify category-specific vocabulary.
- Stopword Filtering - Removes common non-informative words from Chinese text to improve the quality of downstream NLP analysis.
- Keyword and Phrase Extraction - Isolates primary keywords and key phrases from documents to generate concise summaries of main ideas.
- Natural Language Date Parsing - Converts natural language descriptions of dates and holidays into standardized machine-readable timestamps.
- Currency Value Extraction - Parses currency amounts from unstructured text to provide standardized monetary values and currency types.
- Text Data Augmentation - Generates synthetic training examples using synonym replacement and back-translation.
- Chinese Text Augmentation - Creates synthetic training examples through back-translation and synonym replacement to improve Chinese language models.
- National ID Analysis - Extracts demographic and geographic information from Chinese ID numbers to resolve personal identity details.
- Natural Language Processing - All-in-one toolkit for Chinese language processing.
