LuminosoInsight/python-ftfy
Python Ftfy
python-ftfy est une bibliothèque de réparation de texte Unicode conçue pour corriger le mojibake et les problèmes d'encodage. Elle fournit des utilitaires pour la détection d'encodage d'octets, le décodage d'entités HTML et la récupération de texte corrompu pour le restaurer sous sa forme Unicode prévue.
Le projet se distingue par un pipeline de décodage multicouche qui identifie et inverse les mélanges d'encodage complexes. Il utilise une détection basée sur l'heuristique pour résoudre les cas où le texte a été décodé en utilisant le mauvais codec à travers plusieurs couches de corruption, et il peut gérer les variantes UTF-8 non standard et les mappages d'encodage négligents.
La bibliothèque couvre également un large éventail de tâches de normalisation de texte, notamment la normalisation Unicode, la normalisation des sauts de ligne et l'expansion des ligatures latines. Elle inclut des capacités pour la normalisation de la largeur des caractères et la suppression des échappements de terminal et des caractères de contrôle.
Une interface en ligne de commande est disponible pour automatiser la détection et la réparation des problèmes Unicode dans les fichiers.
Features
- Mojibake Detectors - Implements heuristic analysis of character sequences to detect and identify mojibake corruption.
- Mojibake Detection - Implements heuristic-based detection of mojibake and Unicode encoding glitches.
- Multi-Layer Decoders - Implements a multi-layered decoding pipeline to sequentially peel back layers of encoding corruption.
- Character Encoding Detectors - Provides heuristic detection of byte encodings using BOM and UTF-8 validation to identify the correct codec.
- Encoding Recovery Tools - Identifies and reverts multi-layered encoding errors and incorrect codec applications to restore corrupted text.
- Mojibake Repair Utilities - Heuristically detects and fixes mojibake and multi-layered encoding glitches to restore intended text.
- Encoding Mix-up Repair - Provides specialized repair for text corrupted by mismatched single-byte and variable-length encodings.
- Inconsistent Encoding Repair - Detects and fixes text containing fragments of different encodings embedded together.
- Mojibake Restoration - Restores text mangled by incorrect encoding cycles to its intended Unicode representation.
- Unicode Text Repair Libraries - Provides a comprehensive toolset for fixing mojibake and encoding glitches to restore corrupted text.
- Unicode Normalization - Applies standard Unicode normalization forms to ensure characters and combining marks are represented consistently.
- Unicode Text Handling - Fixes encoding glitches and mojibake through configurable transformations applied to text segments.
- Unicode Normalizers - Provides a utility for standardizing line breaks, character widths, and control characters for consistent display.
- Text Cleaning - Cleans Unicode data by removing terminal escapes and decomposing ligatures to prepare text for analysis.
- Escape Sequence Decoding - Converts backslashed escape sequences into their corresponding Unicode characters.
- Line Break Standardization - Converts platform-specific line breaks like CRLF and CR into the standard Unix format.
- Latin Ligature Decomposers - Decomposes single-character Latin ligatures into individual letters to resolve common copy-paste artifacts.
- Lossy Encoding Detection - Identifies UTF-8 decoding errors that resulted in lossy replacement characters.
- Encoding Repair CLIs - Provides a command-line interface to automate the detection and repair of Unicode glitches within files.
- Character Width Normalizers - Replaces halfwidth and fullwidth forms of ASCII, Katakana, and Hangul characters with standard equivalents.
- Transformation Analysis - Offers a detailed list of the specific encoding and decoding steps used to repair a string.
- Non-Standard UTF-8 Decoding - Supports decoding of UTF-8 variants including CESU-8 and Java-style null encodings.
- Encoding Normalizers - Standardizes inconsistent line breaks, character widths, and control characters for consistent display.
- HTML Entity Processors - Converts HTML entity references and backslashed escape sequences into their corresponding Unicode characters.
- Text Processing - Fixes broken Unicode text automatically.
