toolgood/ToolGood.Words
ToolGood.Words
ToolGood.Words is a sensitive word filtering library and text sanitization component designed for high-performance detection and masking of prohibited terms. It provides tools for Chinese text normalization, pinyin transliteration, and the replacement of banned words with placeholders.
The project is distinguished by its ability to uncover obfuscated language through a pinyin transliteration engine and phonetic-based detection. It identifies sensitive content hidden by phonetic substitutions, first-letter initials, or intentional misspellings by mapping Chinese characters to pinyin representations.
The library covers a broad range of text processing capabilities, including character set conversion between simplified and traditional Chinese, full-width and half-width character formatting, and wildcard pattern matching. These utilities support a normalization workflow that standardizes input before applying filtering and masking logic.
Features
- Character-to-Pinyin Converters - Ships a high-performance engine that converts Chinese characters into phonetic pinyin and initials to detect obfuscated language.
- Obfuscation Detection - Identifies sensitive terms that use character skipping, repetitions, or case variations to bypass standard filters.
- Chinese Natural Language Processing - Provides computational linguistics tools for Chinese text, including pinyin transliteration and script transformation.
- Phonetic Representations - Generates pinyin letters or first-letter initials from Chinese characters to uncover hidden sensitive words.
- Sensitive Word Filters - Detects and replaces banned terms in user input with asterisks or custom placeholder messages.
- Chinese Script Normalizers - Standardizes Chinese text by converting between simplified and traditional scripts and adjusting character widths.
- Pinyin Transliterations - Provides a comprehensive system for converting Chinese characters into Pinyin phonetic representations for content moderation.
- Phonetic Dictionaries - Implements dictionary-based mapping of Chinese characters to pinyin to uncover phonetic aliases of banned words.
- Chinese Character Simplifiers - Converts text between Simplified and Traditional Chinese scripts to ensure consistent sensitive word detection.
- Sensitive Text Masking - Implements the replacement of detected sensitive patterns and banned words with placeholders to sanitize content.
- Obfuscated Content Detection - Finds sensitive words hidden by phonetic pinyin substitutions, character repetitions, or intentional misspellings.
- Substring Replacements - Replaces detected sensitive substrings with placeholder characters to sanitize the output text.
- Filter Evasion Normalization - Standardizes text scripts, widths, and case to ensure consistent detection regardless of formatting tricks.
- Variant Character Normalizers - Converts text between traditional and simplified Chinese character sets to ensure consistent detection across writing styles.
- Regex Pattern Matching - Employs regular expressions and wildcards to identify sensitive terms that include character repetitions or spacing.
- Multi-Stage Text Normalizers - Uses a sequential processing chain involving normalization and transliteration to clean and analyze text for sensitive content.
- Pinyin Initialism Matchers - Matches text against a dictionary of pinyin acronyms and initials to catch obfuscated language.
- Pinyin Sequence Pattern Matching - Identifies specific keywords or patterns within pinyin sequences using customizable index and splitting configurations.
- Wildcard Pattern Matching - Identifies sensitive words using partial regular expressions, including dots and question marks.
- Keyword Wildcard Filters - Detects sensitive words using regular expression patterns and wildcards to match various forms of a term.
Star history
How this analysis was created: This summary and feature list were written by an AI model that read the project's README and public documentation pages. Each feature links to the documentation it came from; stars, license and language come straight from the GitHub API. The model does not read the source code, and the analysis is refreshed when the project is re-analysed. Learn more on our About page.
