LuminosoInsight/python-ftfy
Python Ftfy
python-ftfy es una biblioteca de reparación de texto Unicode diseñada para arreglar mojibake y fallos de codificación. Proporciona utilidades para la detección de codificación de bytes, decodificación de entidades HTML y la recuperación de texto corrupto para restaurarlo a su forma Unicode prevista.
El proyecto se distingue a través de un pipeline de decodificación de múltiples capas que identifica y revierte mezclas de codificación complejas. Utiliza detección basada en heurística para resolver casos donde el texto fue decodificado usando el códec incorrecto a través de múltiples capas de corrupción, y puede manejar variantes de UTF-8 no estándar y mapeos de codificación descuidados.
La biblioteca también cubre una amplia gama de tareas de estandarización de texto, incluyendo normalización Unicode, estandarización de saltos de línea y la expansión de ligaduras latinas. Incluye capacidades para la normalización del ancho de caracteres y la eliminación de escapes de terminal y caracteres de control.
Está disponible una interfaz de línea de comandos para automatizar la detección y reparación de fallos Unicode dentro de archivos.
Features
- Mojibake Detectors - Implements heuristic analysis of character sequences to detect and identify mojibake corruption.
- Mojibake Detection - Implements heuristic-based detection of mojibake and Unicode encoding glitches.
- Multi-Layer Decoders - Implements a multi-layered decoding pipeline to sequentially peel back layers of encoding corruption.
- Character Encoding Detectors - Provides heuristic detection of byte encodings using BOM and UTF-8 validation to identify the correct codec.
- Encoding Recovery Tools - Identifies and reverts multi-layered encoding errors and incorrect codec applications to restore corrupted text.
- Mojibake Repair Utilities - Heuristically detects and fixes mojibake and multi-layered encoding glitches to restore intended text.
- Encoding Mix-up Repair - Provides specialized repair for text corrupted by mismatched single-byte and variable-length encodings.
- Inconsistent Encoding Repair - Detects and fixes text containing fragments of different encodings embedded together.
- Mojibake Restoration - Restores text mangled by incorrect encoding cycles to its intended Unicode representation.
- Unicode Text Repair Libraries - Provides a comprehensive toolset for fixing mojibake and encoding glitches to restore corrupted text.
- Unicode Normalization - Applies standard Unicode normalization forms to ensure characters and combining marks are represented consistently.
- Unicode Text Handling - Fixes encoding glitches and mojibake through configurable transformations applied to text segments.
- Unicode Normalizers - Provides a utility for standardizing line breaks, character widths, and control characters for consistent display.
- Text Cleaning - Cleans Unicode data by removing terminal escapes and decomposing ligatures to prepare text for analysis.
- Escape Sequence Decoding - Converts backslashed escape sequences into their corresponding Unicode characters.
- Line Break Standardization - Converts platform-specific line breaks like CRLF and CR into the standard Unix format.
- Latin Ligature Decomposers - Decomposes single-character Latin ligatures into individual letters to resolve common copy-paste artifacts.
- Lossy Encoding Detection - Identifies UTF-8 decoding errors that resulted in lossy replacement characters.
- Encoding Repair CLIs - Provides a command-line interface to automate the detection and repair of Unicode glitches within files.
- Character Width Normalizers - Replaces halfwidth and fullwidth forms of ASCII, Katakana, and Hangul characters with standard equivalents.
- Transformation Analysis - Offers a detailed list of the specific encoding and decoding steps used to repair a string.
- Non-Standard UTF-8 Decoding - Supports decoding of UTF-8 variants including CESU-8 and Java-style null encodings.
- Encoding Normalizers - Standardizes inconsistent line breaks, character widths, and control characters for consistent display.
- HTML Entity Processors - Converts HTML entity references and backslashed escape sequences into their corresponding Unicode characters.
- Text Processing - Fixes broken Unicode text automatically.
