LuminosoInsight/python-ftfy
Python Ftfy
python-ftfy este o bibliotecă de reparare a textului Unicode concepută pentru a remedia mojibake-ul și erorile de codare. Oferă utilitare pentru detectarea codării octeților, decodarea entităților HTML și recuperarea textului corupt pentru a-l restaura la forma Unicode dorită.
Proiectul se distinge printr-un pipeline de decodare pe mai multe straturi care identifică și inversează amestecurile complexe de codare. Utilizează detectarea bazată pe euristică pentru a rezolva cazurile în care textul a fost decodat folosind codec-ul greșit pe mai multe straturi de corupere și poate gestiona variante UTF-8 non-standard și mapări de codare neglijente.
Biblioteca acoperă, de asemenea, o gamă largă de sarcini de standardizare a textului, inclusiv normalizarea Unicode, standardizarea întreruperilor de linie și expansiunea ligaturilor latine. Include capabilități pentru normalizarea lățimii caracterelor și eliminarea escape-urilor de terminal și a caracterelor de control.
O interfață în linie de comandă este disponibilă pentru a automatiza detectarea și repararea erorilor Unicode din fișiere.
Features
- Mojibake Detectors - Implements heuristic analysis of character sequences to detect and identify mojibake corruption.
- Mojibake Detection - Implements heuristic-based detection of mojibake and Unicode encoding glitches.
- Multi-Layer Decoders - Implements a multi-layered decoding pipeline to sequentially peel back layers of encoding corruption.
- Character Encoding Detectors - Provides heuristic detection of byte encodings using BOM and UTF-8 validation to identify the correct codec.
- Encoding Recovery Tools - Identifies and reverts multi-layered encoding errors and incorrect codec applications to restore corrupted text.
- Mojibake Repair Utilities - Heuristically detects and fixes mojibake and multi-layered encoding glitches to restore intended text.
- Encoding Mix-up Repair - Provides specialized repair for text corrupted by mismatched single-byte and variable-length encodings.
- Inconsistent Encoding Repair - Detects and fixes text containing fragments of different encodings embedded together.
- Mojibake Restoration - Restores text mangled by incorrect encoding cycles to its intended Unicode representation.
- Unicode Text Repair Libraries - Provides a comprehensive toolset for fixing mojibake and encoding glitches to restore corrupted text.
- Unicode Normalization - Applies standard Unicode normalization forms to ensure characters and combining marks are represented consistently.
- Unicode Text Handling - Fixes encoding glitches and mojibake through configurable transformations applied to text segments.
- Unicode Normalizers - Provides a utility for standardizing line breaks, character widths, and control characters for consistent display.
- Text Cleaning - Cleans Unicode data by removing terminal escapes and decomposing ligatures to prepare text for analysis.
- Escape Sequence Decoding - Converts backslashed escape sequences into their corresponding Unicode characters.
- Line Break Standardization - Converts platform-specific line breaks like CRLF and CR into the standard Unix format.
- Latin Ligature Decomposers - Decomposes single-character Latin ligatures into individual letters to resolve common copy-paste artifacts.
- Lossy Encoding Detection - Identifies UTF-8 decoding errors that resulted in lossy replacement characters.
- Encoding Repair CLIs - Provides a command-line interface to automate the detection and repair of Unicode glitches within files.
- Character Width Normalizers - Replaces halfwidth and fullwidth forms of ASCII, Katakana, and Hangul characters with standard equivalents.
- Transformation Analysis - Offers a detailed list of the specific encoding and decoding steps used to repair a string.
- Non-Standard UTF-8 Decoding - Supports decoding of UTF-8 variants including CESU-8 and Java-style null encodings.
- Encoding Normalizers - Standardizes inconsistent line breaks, character widths, and control characters for consistent display.
- HTML Entity Processors - Converts HTML entity references and backslashed escape sequences into their corresponding Unicode characters.
- Text Processing - Fixes broken Unicode text automatically.
