# houbb/sensitive-word **Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/houbb-sensitive-word).** 5,896 stars · 790 forks · Java · Apache-2.0 ## Links - GitHub: https://github.com/houbb/sensitive-word - Homepage: https://houbb.github.io/opensource/sensitive-word - awesome-repositories: https://awesome-repositories.com/repository/houbb-sensitive-word.md ## Topics `dfa` `dirty-word` `filter` `java` `nlp` `pinyin` `search` `sensitive` `sensitive-word` `sensitive-word-filter` `string-matching` `textfliter` `trie-tree` ## Description This project is a high-performance Java library and content moderation framework designed to detect and mask prohibited words in text. It utilizes a Deterministic Finite Automaton (DFA) scanner to implement efficient longest-match word detection. The engine distinguishes itself through a text normalization pipeline and noise-filtering preprocessor that standardize character casing, scripts, and widths while removing interspersed special characters to prevent filter evasion. It supports dynamic dictionary management, allowing blacklists and allow-lists to be updated in the background without requiring a system restart. The framework covers a broad range of moderation capabilities, including tag-based word classification for violation categorization, recursive match processing to handle nested sensitive words, and pattern detection for identifying structured data like URLs and email addresses. It also includes utilities for input sanitization to remove HTML tags and script elements. ## Tags ### Data & Databases - [DFA-Based Scanning](https://awesome-repositories.com/f/data-databases/text-pattern-matching/dfa-based-scanning.md) — Implements a high-performance DFA scanner for efficient longest-match sensitive word detection. - [Evasion-Resistant Normalization](https://awesome-repositories.com/f/data-databases/text-normalization/unicode-normalization-pipelines/evasion-resistant-normalization.md) — Standardizes character casing, script styles, and width formats before matching to prevent evasion via formatting tricks. - [Dynamic Blocklist Modification](https://awesome-repositories.com/f/data-databases/data-access-querying/blacklist-filtering/runtime-app-list-blacklist-filters/dynamic-blocklist-modification.md) — Adds or removes specific words from the blacklist without requiring a full re-initialization of the engine. ([source](https://houbb.github.io/2020/01/07/sensitive-word-10-v0.19.0-deny-word-edit)) - [Filter Evasion Normalizers](https://awesome-repositories.com/f/data-databases/text-normalization/filter-evasion-normalizers.md) — Standardizes case, character width, and script styles to prevent filter evasion via formatting tricks. ### Security & Cryptography - [Content Moderation Filters](https://awesome-repositories.com/f/security-cryptography/security-and-access-control/content-moderation-filters.md) — Scans text against configurable blacklists to detect and mask prohibited content in a high-performance framework. ([source](https://houbb.github.io/2020/01/07/sensitive-word-01-intro)) - [Content Cleaning](https://awesome-repositories.com/f/security-cryptography/content-cleaning.md) — Removes URLs, phone numbers, and corrupted characters to isolate the core message for analysis. ([source](https://houbb.github.io/2020/01/07/sensitive-word-03-slim)) - [Dynamic List Updates](https://awesome-repositories.com/f/security-cryptography/data-scrubbing/profanity-scrubbing/word-lists/dynamic-list-updates.md) — Adds or removes words from blacklists and whitelists dynamically to update detection without restarting the system. ([source](https://cdn.jsdelivr.net/gh/houbb/sensitive-word@master/README.md)) - [Dynamic Exception Management](https://awesome-repositories.com/f/security-cryptography/data-scrubbing/profanity-scrubbing/word-lists/false-positive-suppression-lists/dynamic-exception-management.md) — Adds or removes specific words from an exception list dynamically to prevent false positives without restarting. ([source](https://houbb.github.io/2020/01/07/sensitive-word-12-v0.21.0-allow-word-edit)) - [Recursive Filtering](https://awesome-repositories.com/f/security-cryptography/security-and-access-control/content-moderation-filters/recursive-filtering.md) — Scans and filters text repeatedly until no new sensitive words emerge from previous replacements. ([source](https://houbb.github.io/2020/01/07/sensitive-word-01-intro)) - [Middleware-Based Masking](https://awesome-repositories.com/f/security-cryptography/sensitive-data-access-controls/sensitive-content-obscuration/middleware-based-masking.md) — Replaces sensitive words with alternative characters or masks based on custom replacement strategies. ([source](https://github.com/houbb/sensitive-word/blob/master/CHANGE_LOG.md)) - [Violation Categorization](https://awesome-repositories.com/f/security-cryptography/violation-categorization.md) — Assigns labels to detected words to distinguish between different violation types, such as fraud or violence. ([source](https://houbb.github.io/2020/01/07/sensitive-word-01-intro)) ### Artificial Intelligence & ML - [Sensitive Word Filters](https://awesome-repositories.com/f/artificial-intelligence-ml/stop-word-filters/sensitive-word-filters.md) — Provides a noise-filtering preprocessor to detect banned terms hidden by interspersed special characters or filler symbols. - [Dictionary Management](https://awesome-repositories.com/f/artificial-intelligence-ml/stop-word-filters/sensitive-word-filters/dictionary-management.md) — Provides administrative management for sensitive word dictionaries and allow-lists to tailor prohibited content detection. ([source](https://github.com/houbb/sensitive-word/blob/master/CHANGE_LOG.md)) - [Noise Character Filtering](https://awesome-repositories.com/f/artificial-intelligence-ml/stop-word-filters/sensitive-word-filters/noise-character-filtering.md) — Ignores specific characters or stop words during scanning to find sensitive terms hidden by filler symbols. ([source](https://houbb.github.io/2020/01/07/sensitive-word-01-intro)) - [Permutation Analysis](https://awesome-repositories.com/f/artificial-intelligence-ml/stop-word-filters/sensitive-word-filters/permutation-analysis.md) — Analyzes reversed character sequences and phonetic variations to detect sensitive words intended to bypass filters. ([source](https://houbb.github.io/2020/01/07/sensitive-word-03-slim)) ### Part of an Awesome List - [Input Validation and Sanitization](https://awesome-repositories.com/f/awesome-lists/security/input-validation-and-sanitization.md) — Provides libraries for validating, masking, and sanitizing data to prevent offensive content. - [Data Detection Patterns](https://awesome-repositories.com/f/awesome-lists/devtools/regex-and-pattern-matching/data-detection-patterns.md) — Combines dictionary lookups with regular expressions to detect structured data such as URLs, emails, and IP addresses. ([source](https://houbb.github.io/2020/01/07/sensitive-word-14-v0.25.0-url-define)) ### Software Engineering & Architecture - [Finite Automaton Search](https://awesome-repositories.com/f/software-engineering-architecture/finite-automaton-search.md) — Utilizes a Deterministic Finite Automaton for high-performance pattern matching and sensitive word detection. - [Finite Automaton Search Algorithms](https://awesome-repositories.com/f/software-engineering-architecture/finite-automaton-search-algorithms.md) — Implements high-performance keyword matching in Java using the Deterministic Finite Automaton algorithm for fast content scanning. - [Longest-Match String Resolution](https://awesome-repositories.com/f/software-engineering-architecture/string-processing-algorithms/longest-common-substring-algorithms/longest-match-string-resolution.md) — Prioritizes the longest matching sequence from the dictionary to ensure precise identification of complex sensitive phrases. - [Filter Evasion Normalization](https://awesome-repositories.com/f/software-engineering-architecture/string-validation-and-normalization/speech-to-text-normalizers/character-width-normalizers/filter-evasion-normalization.md) — Standardizes text scripts, character widths, and case to prevent users from bypassing filters with formatting tricks. - [Filter Evasion Normalization](https://awesome-repositories.com/f/software-engineering-architecture/string-validation-and-normalization/speech-to-text-normalizers/custom-text-normalizers/filter-evasion-normalization.md) — Standardizes case, character width, and script styles to prevent users from bypassing filters with formatting tricks. ([source](https://github.com/houbb/sensitive-word)) - [Longest-Match Word Detection](https://awesome-repositories.com/f/software-engineering-architecture/suffix-trees/longest-match-word-detection.md) — Implements a high-precision longest-match algorithm using a DFA to identify prohibited words in text. ([source](https://cdn.jsdelivr.net/gh/houbb/sensitive-word@master/README.md)) - [False Positive Filtering](https://awesome-repositories.com/f/software-engineering-architecture/false-positive-filtering.md) — Reduces false positives using custom conditions, such as category tags or full-word requirements. ([source](https://houbb.github.io/2020/01/07/sensitive-word-13-v0.23.0-result-condition-enhance)) - [Recursive Replacement Loops](https://awesome-repositories.com/f/software-engineering-architecture/recursive-replacement-loops.md) — Scans and filters text repeatedly until no new sensitive words are formed by previous replacement operations. - [Dictionary Background Refresh](https://awesome-repositories.com/f/software-engineering-architecture/service-discovery-registries/registry-client-caching/dictionary-background-refresh.md) — Updates the active sensitive word library in the background to maintain current lists without interrupting existing operations. ([source](https://houbb.github.io/2020/01/07/sensitive-word-00-overview)) ### Business & Productivity Software - [Violation Category Tagging](https://awesome-repositories.com/f/business-productivity-software/tag-filtering-systems/label-based-filtering/violation-category-tagging.md) — Associates metadata labels with dictionary entries to enable filtering and processing based on specific violation categories. ### Development Tools & Productivity - [Dictionary Hot-Reloading](https://awesome-repositories.com/f/development-tools-productivity/dynamic-logic-reloading/dictionary-hot-reloading.md) — Allows updating sensitive word blacklists and allow-lists in the background without requiring a system restart. ### Programming Languages & Runtimes - [Intervening Character Filtering](https://awesome-repositories.com/f/programming-languages-runtimes/string-sanitizers/character-whitelist-and-blacklist-filters/intervening-character-filtering.md) — Filters out specified special characters during scanning to find words intentionally separated by noise. ([source](https://houbb.github.io/2020/01/07/sensitive-word-05-ignore-char))