rspeer/python-ftfy

Features

• Mojibake Repair Utilities - Repairs complex, multi-layered encoding errors and mojibake patterns to restore the original intended text.
• Unicode Text Repair Libraries - Provides a comprehensive library for repairing mojibake and encoding glitches to restore intended Unicode text.
• Text Cleaning - Cleans text data by removing invisible control characters and terminal escapes while standardizing ligatures.
• Byte Encoding Detectors - Identifies the likely encoding of byte strings by checking for byte-order marks and UTF-8 validity.
• Character Encoding Detectors - Analyzes byte sequences to identify the most likely character encoding and detect lossy sequences.
• Lossy Encoding Detection - Identifies lossy encoding sequences, such as replacement characters, to determine how text was mangled.
• Inconsistent Encoding Repair - Detects and resolves instances where multiple different encodings are embedded within a single text stream.
• Mojibake Detection - Uses character sequence heuristics to identify likely Unicode encoding glitches and mojibake.
• Mojibake Restoration - Reverses multi-layered encoding errors to restore original characters from mangled UTF-8 and single-byte strings.
• Unicode Normalization - Standardizes character widths and combining marks to ensure consistent string representations across platforms.
• Recursive Encoding Reversals - Fixes multi-layered encoding errors by recursively applying decoding and encoding cycles until the text is stable.
• Unicode Normalizers - Standardizes UTF-8 text through character decomposition, width normalization, and resolving Latin ligatures.
• Multi-Stage Text Normalizers - Processes text through a sequence of cleaning, decoding, and normalizing steps to resolve mixed encoding glitches.
• Surrogate Pair Correctors - Replaces UTF-16 surrogate pairs with correct characters to fix text decoded via obsolete standards.
• Sloppy Encoding Mapping - Implements mapping of unassigned bytes in single-byte encodings to compatible Unicode codepoints for legacy browser interoperability.
• Byte Order Mark Detectors - Provides a mechanism to guess original text encoding by checking for signature byte-order marks.
• Encoding Error Analyzers - Analyzes strings to identify specific encoding errors and lists the transformations required to fix the text.
• Escape Sequence Decoding - Converts hex and Unicode backslashed escape sequences into their corresponding Unicode characters.
• Line Break Standardization - Standardizes platform-specific newline sequences like CRLF and CR into Unix-style line breaks.
• Ligature Expansion - Decomposes single-character Latin ligatures into the individual letters they represent to fix copy-paste artifacts.
• Encoding Repair CLIs - Provides a command-line utility for detecting and repairing mojibake and encoding glitches in files.
• Unicode Character Inspectors - Analyzes Unicode strings by displaying codepoints, hexadecimal values, glyphs, and character categories.
• Character Substitution Tables - Employs predefined dictionaries to replace ligatures and non-standard control characters with standard equivalents.
• Character Width Normalizers - Replaces halfwidth and fullwidth forms of ASCII, katakana, and Hangul with standard Unicode representations.
• Control Character Normalizers - Provides utilities that map C1 control characters to Windows-1252 equivalents to ensure web-standard compatibility.
• Non-Standard UTF-8 Decoding - Decodes non-standard UTF-8 variants including CESU-8 and Java-specific null character encodings.
• HTML Entity Processors - Converts HTML entity sequences and backslashed escapes into their corresponding Unicode characters.
• Natural Language Processing - Utility for fixing Unicode glitches and mojibake.
• General Utilities - Tool for fixing broken Unicode strings.
• Text Processing - Fixes broken Unicode text to make it consistent.

Star history