LuminosoInsight/python-ftfy
Python Ftfy
python-ftfy is a Unicode text repair library designed to fix mojibake and encoding glitches. It provides utilities for byte encoding detection, HTML entity decoding, and the recovery of corrupted text to restore it to its intended Unicode form.
The project distinguishes itself through a multi-layered decoding pipeline that identifies and reverts complex encoding mix-ups. It uses heuristic-based detection to resolve instances where text was decoded using the wrong codec across multiple layers of corruption, and it can handle non-standard UTF-8 variants and sloppy encoding mappings.
The library also covers a broad range of text standardization tasks, including Unicode normalization, line break standardization, and the expansion of Latin ligatures. It includes capabilities for character width normalization and the removal of terminal escapes and control characters.
A command-line interface is available to automate the detection and repair of Unicode glitches within files.
Features
- Mojibake Detectors - Implements heuristic analysis of character sequences to detect and identify mojibake corruption.
- Mojibake Detection - Implements heuristic-based detection of mojibake and Unicode encoding glitches.
- Multi-Layer Decoders - Implements a multi-layered decoding pipeline to sequentially peel back layers of encoding corruption.
- Character Encoding Detectors - Provides heuristic detection of byte encodings using BOM and UTF-8 validation to identify the correct codec.
- Encoding Recovery Tools - Identifies and reverts multi-layered encoding errors and incorrect codec applications to restore corrupted text.
- Mojibake Repair Utilities - Heuristically detects and fixes mojibake and multi-layered encoding glitches to restore intended text.
- Encoding Mix-up Repair - Provides specialized repair for text corrupted by mismatched single-byte and variable-length encodings.
- Inconsistent Encoding Repair - Detects and fixes text containing fragments of different encodings embedded together.
- Mojibake Restoration - Restores text mangled by incorrect encoding cycles to its intended Unicode representation.
- Unicode Text Repair Libraries - Provides a comprehensive toolset for fixing mojibake and encoding glitches to restore corrupted text.
- Unicode Normalization - Applies standard Unicode normalization forms to ensure characters and combining marks are represented consistently.
- Unicode Text Handling - Fixes encoding glitches and mojibake through configurable transformations applied to text segments.
- Unicode Normalizers - Provides a utility for standardizing line breaks, character widths, and control characters for consistent display.
- Text Cleaning - Cleans Unicode data by removing terminal escapes and decomposing ligatures to prepare text for analysis.
- Escape Sequence Decoding - Converts backslashed escape sequences into their corresponding Unicode characters.
- Line Break Standardization - Converts platform-specific line breaks like CRLF and CR into the standard Unix format.
- Latin Ligature Decomposers - Decomposes single-character Latin ligatures into individual letters to resolve common copy-paste artifacts.
- Lossy Encoding Detection - Identifies UTF-8 decoding errors that resulted in lossy replacement characters.
- Encoding Repair CLIs - Provides a command-line interface to automate the detection and repair of Unicode glitches within files.
- Character Width Normalizers - Replaces halfwidth and fullwidth forms of ASCII, Katakana, and Hangul characters with standard equivalents.
- Transformation Analysis - Offers a detailed list of the specific encoding and decoding steps used to repair a string.
- Non-Standard UTF-8 Decoding - Supports decoding of UTF-8 variants including CESU-8 and Java-style null encodings.
- Encoding Normalizers - Standardizes inconsistent line breaks, character widths, and control characters for consistent display.
- HTML Entity Processors - Converts HTML entity references and backslashed escape sequences into their corresponding Unicode characters.
- Text Processing - Fixes broken Unicode text automatically.
