toolgood/ToolGood.Words
ToolGood.Words
ToolGood.Words es una librería de filtrado de palabras sensibles y un componente de sanitización de texto diseñado para la detección y enmascaramiento de alto rendimiento de términos prohibidos. Proporciona herramientas para la normalización de texto en chino, transliteración pinyin y el reemplazo de palabras prohibidas con marcadores de posición.
El proyecto se distingue por su capacidad para descubrir lenguaje ofuscado a través de un motor de transliteración pinyin y detección basada en fonética. Identifica contenido sensible oculto mediante sustituciones fonéticas, iniciales de primera letra o errores ortográficos intencionales al mapear caracteres chinos a representaciones pinyin.
La librería cubre una amplia gama de capacidades de procesamiento de texto, incluyendo conversión de conjuntos de caracteres entre chino simplificado y tradicional, formato de caracteres de ancho completo y medio, y coincidencia de patrones con comodines. Estas utilidades soportan un flujo de trabajo de normalización que estandariza la entrada antes de aplicar la lógica de filtrado y enmascaramiento.
Features
- Character-to-Pinyin Converters - Ships a high-performance engine that converts Chinese characters into phonetic pinyin and initials to detect obfuscated language.
- Obfuscation Detection - Identifies sensitive terms that use character skipping, repetitions, or case variations to bypass standard filters.
- Chinese Natural Language Processing - Provides computational linguistics tools for Chinese text, including pinyin transliteration and script transformation.
- Phonetic Representations - Generates pinyin letters or first-letter initials from Chinese characters to uncover hidden sensitive words.
- Sensitive Word Filters - Detects and replaces banned terms in user input with asterisks or custom placeholder messages.
- Chinese Script Normalizers - Standardizes Chinese text by converting between simplified and traditional scripts and adjusting character widths.
- Pinyin Transliterations - Provides a comprehensive system for converting Chinese characters into Pinyin phonetic representations for content moderation.
- Phonetic Dictionaries - Implements dictionary-based mapping of Chinese characters to pinyin to uncover phonetic aliases of banned words.
- Chinese Character Simplifiers - Converts text between Simplified and Traditional Chinese scripts to ensure consistent sensitive word detection.
- Sensitive Text Masking - Implements the replacement of detected sensitive patterns and banned words with placeholders to sanitize content.
- Obfuscated Content Detection - Finds sensitive words hidden by phonetic pinyin substitutions, character repetitions, or intentional misspellings.
- Substring Replacements - Replaces detected sensitive substrings with placeholder characters to sanitize the output text.
- Filter Evasion Normalization - Standardizes text scripts, widths, and case to ensure consistent detection regardless of formatting tricks.
- Variant Character Normalizers - Converts text between traditional and simplified Chinese character sets to ensure consistent detection across writing styles.
- Regex Pattern Matching - Employs regular expressions and wildcards to identify sensitive terms that include character repetitions or spacing.
- Multi-Stage Text Normalizers - Uses a sequential processing chain involving normalization and transliteration to clean and analyze text for sensitive content.
- Pinyin Initialism Matchers - Matches text against a dictionary of pinyin acronyms and initials to catch obfuscated language.
- Pinyin Sequence Pattern Matching - Identifies specific keywords or patterns within pinyin sequences using customizable index and splitting configurations.
- Wildcard Pattern Matching - Identifies sensitive words using partial regular expressions, including dots and question marks.
- Keyword Wildcard Filters - Detects sensitive words using regular expression patterns and wildcards to match various forms of a term.
