rspeer/python-ftfy
Python Ftfy
This project is a Unicode text repair tool and mojibake correction library designed to fix encoding glitches and restore original characters from mangled strings. It functions as a text encoding detector and a Unicode normalization tool to resolve issues where text has been incorrectly decoded.
The library specializes in reversing multi-layered encoding errors and repairing complex mojibake patterns. It includes capabilities for detecting lossy encoding sequences, guessing byte encodings, and decoding non-standard UTF-8 variants.
The toolset covers a broad range of text cleaning and normalization tasks, including the decoding of HTML entities and backslashed escapes, the expansion of Latin ligatures, and the standardization of character widths and line breaks. It also provides utilities for removing invisible control characters and inspecting Unicode strings by codepoint.
A command-line interface is available for repairing Unicode glitches and encoding errors within files or input streams.
Features
- Mojibake Repair Utilities - Repairs complex, multi-layered encoding errors and mojibake patterns to restore the original intended text.
- Unicode Text Repair Libraries - Provides a comprehensive library for repairing mojibake and encoding glitches to restore intended Unicode text.
- Text Cleaning - Cleans text data by removing invisible control characters and terminal escapes while standardizing ligatures.
- Byte Encoding Detectors - Identifies the likely encoding of byte strings by checking for byte-order marks and UTF-8 validity.
- Character Encoding Detectors - Analyzes byte sequences to identify the most likely character encoding and detect lossy sequences.
- Lossy Encoding Detection - Identifies lossy encoding sequences, such as replacement characters, to determine how text was mangled.
- Inconsistent Encoding Repair - Detects and resolves instances where multiple different encodings are embedded within a single text stream.
- Mojibake Detection - Uses character sequence heuristics to identify likely Unicode encoding glitches and mojibake.
- Mojibake Restoration - Reverses multi-layered encoding errors to restore original characters from mangled UTF-8 and single-byte strings.
- Unicode Normalization - Standardizes character widths and combining marks to ensure consistent string representations across platforms.
- Recursive Encoding Reversals - Fixes multi-layered encoding errors by recursively applying decoding and encoding cycles until the text is stable.
- Unicode Normalizers - Standardizes UTF-8 text through character decomposition, width normalization, and resolving Latin ligatures.
- Multi-Stage Text Normalizers - Processes text through a sequence of cleaning, decoding, and normalizing steps to resolve mixed encoding glitches.
- Surrogate Pair Correctors - Replaces UTF-16 surrogate pairs with correct characters to fix text decoded via obsolete standards.
- Sloppy Encoding Mapping - Implements mapping of unassigned bytes in single-byte encodings to compatible Unicode codepoints for legacy browser interoperability.
- Byte Order Mark Detectors - Provides a mechanism to guess original text encoding by checking for signature byte-order marks.
- Encoding Error Analyzers - Analyzes strings to identify specific encoding errors and lists the transformations required to fix the text.
- Escape Sequence Decoding - Converts hex and Unicode backslashed escape sequences into their corresponding Unicode characters.
- Line Break Standardization - Standardizes platform-specific newline sequences like CRLF and CR into Unix-style line breaks.
- Ligature Expansion - Decomposes single-character Latin ligatures into the individual letters they represent to fix copy-paste artifacts.
- Encoding Repair CLIs - Provides a command-line utility for detecting and repairing mojibake and encoding glitches in files.
- Unicode Character Inspectors - Analyzes Unicode strings by displaying codepoints, hexadecimal values, glyphs, and character categories.
- Character Substitution Tables - Employs predefined dictionaries to replace ligatures and non-standard control characters with standard equivalents.
- Character Width Normalizers - Replaces halfwidth and fullwidth forms of ASCII, katakana, and Hangul with standard Unicode representations.
- Control Character Normalizers - Provides utilities that map C1 control characters to Windows-1252 equivalents to ensure web-standard compatibility.
- Non-Standard UTF-8 Decoding - Decodes non-standard UTF-8 variants including CESU-8 and Java-specific null character encodings.
- HTML Entity Processors - Converts HTML entity sequences and backslashed escapes into their corresponding Unicode characters.
- Natural Language Processing - Utility for fixing Unicode glitches and mojibake.
- General Utilities - Tool for fixing broken Unicode strings.
- Text Processing - Fixes broken Unicode text to make it consistent.
