LuminosoInsight/python-ftfy

Features

• Mojibake Detectors - Implements heuristic analysis of character sequences to detect and identify mojibake corruption.
• Mojibake Detection - Implements heuristic-based detection of mojibake and Unicode encoding glitches.
• Multi-Layer Decoders - Implements a multi-layered decoding pipeline to sequentially peel back layers of encoding corruption.
• Character Encoding Detectors - Provides heuristic detection of byte encodings using BOM and UTF-8 validation to identify the correct codec.
• Encoding Recovery Tools - Identifies and reverts multi-layered encoding errors and incorrect codec applications to restore corrupted text.
• Mojibake Repair Utilities - Heuristically detects and fixes mojibake and multi-layered encoding glitches to restore intended text.
• Encoding Mix-up Repair - Provides specialized repair for text corrupted by mismatched single-byte and variable-length encodings.
• Inconsistent Encoding Repair - Detects and fixes text containing fragments of different encodings embedded together.
• Mojibake Restoration - Restores text mangled by incorrect encoding cycles to its intended Unicode representation.
• Unicode Text Repair Libraries - Provides a comprehensive toolset for fixing mojibake and encoding glitches to restore corrupted text.
• Unicode Normalization - Applies standard Unicode normalization forms to ensure characters and combining marks are represented consistently.
• Unicode Text Handling - Fixes encoding glitches and mojibake through configurable transformations applied to text segments.
• Unicode Normalizers - Provides a utility for standardizing line breaks, character widths, and control characters for consistent display.
• Text Cleaning - Cleans Unicode data by removing terminal escapes and decomposing ligatures to prepare text for analysis.
• Escape Sequence Decoding - Converts backslashed escape sequences into their corresponding Unicode characters.
• Line Break Standardization - Converts platform-specific line breaks like CRLF and CR into the standard Unix format.
• Latin Ligature Decomposers - Decomposes single-character Latin ligatures into individual letters to resolve common copy-paste artifacts.
• Lossy Encoding Detection - Identifies UTF-8 decoding errors that resulted in lossy replacement characters.
• Encoding Repair CLIs - Provides a command-line interface to automate the detection and repair of Unicode glitches within files.
• Character Width Normalizers - Replaces halfwidth and fullwidth forms of ASCII, Katakana, and Hangul characters with standard equivalents.
• Transformation Analysis - Offers a detailed list of the specific encoding and decoding steps used to repair a string.
• Non-Standard UTF-8 Decoding - Supports decoding of UTF-8 variants including CESU-8 and Java-style null encodings.
• Encoding Normalizers - Standardizes inconsistent line breaks, character widths, and control characters for consistent display.
• HTML Entity Processors - Converts HTML entity references and backslashed escape sequences into their corresponding Unicode characters.
• Text Processing - Fixes broken Unicode text automatically.

Star history