houbb/sensitive-word

Features

• DFA-Based Scanning - Implements a high-performance DFA scanner for efficient longest-match sensitive word detection.
• Content Moderation Filters - Scans text against configurable blacklists to detect and mask prohibited content in a high-performance framework.
• Sensitive Word Filters - Provides a noise-filtering preprocessor to detect banned terms hidden by interspersed special characters or filler symbols.
• Input Validation and Sanitization - Provides libraries for validating, masking, and sanitizing data to prevent offensive content.
• Evasion-Resistant Normalization - Standardizes character casing, script styles, and width formats before matching to prevent evasion via formatting tricks.
• Finite Automaton Search - Utilizes a Deterministic Finite Automaton for high-performance pattern matching and sensitive word detection.
• Finite Automaton Search Algorithms - Implements high-performance keyword matching in Java using the Deterministic Finite Automaton algorithm for fast content scanning.
• Longest-Match String Resolution - Prioritizes the longest matching sequence from the dictionary to ensure precise identification of complex sensitive phrases.
• Filter Evasion Normalization - Standardizes text scripts, character widths, and case to prevent users from bypassing filters with formatting tricks.
• Filter Evasion Normalization - Standardizes case, character width, and script styles to prevent users from bypassing filters with formatting tricks.
• Longest-Match Word Detection - Implements a high-precision longest-match algorithm using a DFA to identify prohibited words in text.
• Dictionary Management - Provides administrative management for sensitive word dictionaries and allow-lists to tailor prohibited content detection.
• Noise Character Filtering - Ignores specific characters or stop words during scanning to find sensitive terms hidden by filler symbols.
• Permutation Analysis - Analyzes reversed character sequences and phonetic variations to detect sensitive words intended to bypass filters.
• Data Detection Patterns - Combines dictionary lookups with regular expressions to detect structured data such as URLs, emails, and IP addresses.
• Violation Category Tagging - Associates metadata labels with dictionary entries to enable filtering and processing based on specific violation categories.
• Dynamic Blocklist Modification - Adds or removes specific words from the blacklist without requiring a full re-initialization of the engine.
• Filter Evasion Normalizers - Standardizes case, character width, and script styles to prevent filter evasion via formatting tricks.
• Dictionary Hot-Reloading - Allows updating sensitive word blacklists and allow-lists in the background without requiring a system restart.
• Intervening Character Filtering - Filters out specified special characters during scanning to find words intentionally separated by noise.
• Content Cleaning - Removes URLs, phone numbers, and corrupted characters to isolate the core message for analysis.
• Dynamic List Updates - Adds or removes words from blacklists and whitelists dynamically to update detection without restarting the system.
• Dynamic Exception Management - Adds or removes specific words from an exception list dynamically to prevent false positives without restarting.
• Recursive Filtering - Scans and filters text repeatedly until no new sensitive words emerge from previous replacements.
• Middleware-Based Masking - Replaces sensitive words with alternative characters or masks based on custom replacement strategies.
• Violation Categorization - Assigns labels to detected words to distinguish between different violation types, such as fraud or violence.
• False Positive Filtering - Reduces false positives using custom conditions, such as category tags or full-word requirements.
• Recursive Replacement Loops - Scans and filters text repeatedly until no new sensitive words are formed by previous replacement operations.
• Dictionary Background Refresh - Updates the active sensitive word library in the background to maintain current lists without interrupting existing operations.

Star history