rspeer/python-ftfy
Python Ftfy
Ce projet est un outil de réparation de texte Unicode et une bibliothèque de correction de mojibake conçus pour corriger les problèmes d'encodage et restaurer les caractères originaux à partir de chaînes corrompues. Il fonctionne comme un détecteur d'encodage de texte et un outil de normalisation Unicode pour résoudre les problèmes où le texte a été incorrectement décodé.
La bibliothèque se spécialise dans l'inversion des erreurs d'encodage multicouches et la réparation des modèles de mojibake complexes. Elle inclut des capacités pour détecter les séquences d'encodage avec perte, deviner les encodages d'octets et décoder les variantes UTF-8 non standard.
L'ensemble d'outils couvre un large éventail de tâches de nettoyage et de normalisation de texte, notamment le décodage des entités HTML et des échappements par barre oblique inverse, l'expansion des ligatures latines et la normalisation des largeurs de caractères et des sauts de ligne. Il fournit également des utilitaires pour supprimer les caractères de contrôle invisibles et inspecter les chaînes Unicode par point de code.
Une interface en ligne de commande est disponible pour réparer les problèmes Unicode et les erreurs d'encodage dans les fichiers ou les flux d'entrée.
Features
- Mojibake Repair Utilities - Repairs complex, multi-layered encoding errors and mojibake patterns to restore the original intended text.
- Unicode Text Repair Libraries - Provides a comprehensive library for repairing mojibake and encoding glitches to restore intended Unicode text.
- Text Cleaning - Cleans text data by removing invisible control characters and terminal escapes while standardizing ligatures.
- Byte Encoding Detectors - Identifies the likely encoding of byte strings by checking for byte-order marks and UTF-8 validity.
- Character Encoding Detectors - Analyzes byte sequences to identify the most likely character encoding and detect lossy sequences.
- Lossy Encoding Detection - Identifies lossy encoding sequences, such as replacement characters, to determine how text was mangled.
- Inconsistent Encoding Repair - Detects and resolves instances where multiple different encodings are embedded within a single text stream.
- Mojibake Detection - Uses character sequence heuristics to identify likely Unicode encoding glitches and mojibake.
- Mojibake Restoration - Reverses multi-layered encoding errors to restore original characters from mangled UTF-8 and single-byte strings.
- Unicode Normalization - Standardizes character widths and combining marks to ensure consistent string representations across platforms.
- Recursive Encoding Reversals - Fixes multi-layered encoding errors by recursively applying decoding and encoding cycles until the text is stable.
- Unicode Normalizers - Standardizes UTF-8 text through character decomposition, width normalization, and resolving Latin ligatures.
- Multi-Stage Text Normalizers - Processes text through a sequence of cleaning, decoding, and normalizing steps to resolve mixed encoding glitches.
- Surrogate Pair Correctors - Replaces UTF-16 surrogate pairs with correct characters to fix text decoded via obsolete standards.
- Sloppy Encoding Mapping - Implements mapping of unassigned bytes in single-byte encodings to compatible Unicode codepoints for legacy browser interoperability.
- Byte Order Mark Detectors - Provides a mechanism to guess original text encoding by checking for signature byte-order marks.
- Encoding Error Analyzers - Analyzes strings to identify specific encoding errors and lists the transformations required to fix the text.
- Escape Sequence Decoding - Converts hex and Unicode backslashed escape sequences into their corresponding Unicode characters.
- Line Break Standardization - Standardizes platform-specific newline sequences like CRLF and CR into Unix-style line breaks.
- Ligature Expansion - Decomposes single-character Latin ligatures into the individual letters they represent to fix copy-paste artifacts.
- Encoding Repair CLIs - Provides a command-line utility for detecting and repairing mojibake and encoding glitches in files.
- Unicode Character Inspectors - Analyzes Unicode strings by displaying codepoints, hexadecimal values, glyphs, and character categories.
- Character Substitution Tables - Employs predefined dictionaries to replace ligatures and non-standard control characters with standard equivalents.
- Character Width Normalizers - Replaces halfwidth and fullwidth forms of ASCII, katakana, and Hangul with standard Unicode representations.
- Control Character Normalizers - Provides utilities that map C1 control characters to Windows-1252 equivalents to ensure web-standard compatibility.
- Non-Standard UTF-8 Decoding - Decodes non-standard UTF-8 variants including CESU-8 and Java-specific null character encodings.
- HTML Entity Processors - Converts HTML entity sequences and backslashed escapes into their corresponding Unicode characters.
- Natural Language Processing - Utility for fixing Unicode glitches and mojibake.
- General Utilities - Tool for fixing broken Unicode strings.
- Text Processing - Fixes broken Unicode text to make it consistent.
