toolgood/ToolGood.Words
ToolGood.Words
ToolGood.Words ist eine Bibliothek zur Filterung sensibler Wörter und eine Komponente zur Textbereinigung, die für die performante Erkennung und Maskierung verbotener Begriffe entwickelt wurde. Sie bietet Tools für die Normalisierung chinesischer Texte, Pinyin-Transliteration und das Ersetzen gesperrter Wörter durch Platzhalter.
Das Projekt zeichnet sich durch seine Fähigkeit aus, verschleierte Sprache mittels einer Pinyin-Transliterations-Engine und phonetischer Erkennung aufzudecken. Es identifiziert sensible Inhalte, die durch phonetische Ersetzungen, Initialen oder absichtliche Rechtschreibfehler versteckt wurden, indem chinesische Schriftzeichen in Pinyin-Repräsentationen umgewandelt werden.
Die Bibliothek deckt ein breites Spektrum an Textverarbeitungsfunktionen ab, einschließlich Zeichensatzkonvertierung zwischen vereinfachtem und traditionellem Chinesisch, Formatierung von Voll- und Halbbreitenzeichen sowie Wildcard-Musterabgleich. Diese Utilities unterstützen einen Normalisierungs-Workflow, der Eingaben standardisiert, bevor Filter- und Maskierungslogik angewendet wird.
Features
- Character-to-Pinyin Converters - Ships a high-performance engine that converts Chinese characters into phonetic pinyin and initials to detect obfuscated language.
- Obfuscation Detection - Identifies sensitive terms that use character skipping, repetitions, or case variations to bypass standard filters.
- Chinese Natural Language Processing - Provides computational linguistics tools for Chinese text, including pinyin transliteration and script transformation.
- Phonetic Representations - Generates pinyin letters or first-letter initials from Chinese characters to uncover hidden sensitive words.
- Sensitive Word Filters - Detects and replaces banned terms in user input with asterisks or custom placeholder messages.
- Chinese Script Normalizers - Standardizes Chinese text by converting between simplified and traditional scripts and adjusting character widths.
- Pinyin Transliterations - Provides a comprehensive system for converting Chinese characters into Pinyin phonetic representations for content moderation.
- Phonetic Dictionaries - Implements dictionary-based mapping of Chinese characters to pinyin to uncover phonetic aliases of banned words.
- Chinese Character Simplifiers - Converts text between Simplified and Traditional Chinese scripts to ensure consistent sensitive word detection.
- Sensitive Text Masking - Implements the replacement of detected sensitive patterns and banned words with placeholders to sanitize content.
- Obfuscated Content Detection - Finds sensitive words hidden by phonetic pinyin substitutions, character repetitions, or intentional misspellings.
- Substring Replacements - Replaces detected sensitive substrings with placeholder characters to sanitize the output text.
- Filter Evasion Normalization - Standardizes text scripts, widths, and case to ensure consistent detection regardless of formatting tricks.
- Variant Character Normalizers - Converts text between traditional and simplified Chinese character sets to ensure consistent detection across writing styles.
- Regex Pattern Matching - Employs regular expressions and wildcards to identify sensitive terms that include character repetitions or spacing.
- Multi-Stage Text Normalizers - Uses a sequential processing chain involving normalization and transliteration to clean and analyze text for sensitive content.
- Pinyin Initialism Matchers - Matches text against a dictionary of pinyin acronyms and initials to catch obfuscated language.
- Pinyin Sequence Pattern Matching - Identifies specific keywords or patterns within pinyin sequences using customizable index and splitting configurations.
- Wildcard Pattern Matching - Identifies sensitive words using partial regular expressions, including dots and question marks.
- Keyword Wildcard Filters - Detects sensitive words using regular expression patterns and wildcards to match various forms of a term.
