shibing624/pycorrector

Features

• Error Correction Toolkits - Provides a comprehensive open-source toolkit for detecting and correcting spelling and grammar errors in Chinese text.
• Confusion Set Matchers - Replaces characters or words by looking them up in hand-curated phonetic, shape-similar, or custom mapping tables.
• Chinese Error Correction Models - Ships a fine-tuned BERT model for detecting and correcting character-level spelling errors in Chinese text.
• N-Gram Language Models - Scores candidate corrections by their perplexity under a Kenlm n-gram language model to rank and select the best fix.
• Chinese Spelling Correctors - Provides a rule-based corrector using Kenlm n-gram language models for phonetic and shape-similar character errors.
• Phonetic Homophone Correction - Replaces characters that sound alike but are wrong using a confusion set of homophones to fix common misspellings.
• Perplexity-Based Error Detectors - Detects erroneous character positions by comparing character-level perplexity scores against a tunable threshold from a language model.
• Chinese Spelling Correctors - Corrects Chinese spelling mistakes using statistical and neural models for phonetic and shape-similar errors.
• Shape-Similar Error Correctors - Replaces characters that look alike but are wrong using a confusion set of visually similar glyphs.
• Text Error Detection APIs - Provides APIs that identify error positions and types in text without performing correction.
• Chinese Error Detection APIs - Implements APIs that detect erroneous characters in Chinese text and return their positions and error types.
• Chinese Error Correction Models - Ships GPT-based correctors (ChatGLM3, Qwen2.5) for detecting and fixing phonetic and grammatical errors in Chinese.
• Chinese Error Correction Models - Uses a fine-tuned BERT model to predict and correct character-level errors in Chinese text input.
• Transformer Fine-Tuning - Fine-tunes encoder-decoder or decoder-only transformer models on paired error-correction data for sequence-level text correction.
• Chinese Error Correction Models - Ships a T5 sequence-to-sequence corrector for handling length-mismatched grammar errors in Chinese text.
• Deep Learning Correctors - Employs end-to-end deep learning models like RNN, CRF, and transformers to automatically fix errors in Chinese text.
• Built-In Language Model Correctors - Corrects Chinese text errors using built-in language models for phonetic, shape-similar, and grammatical mistakes.
• Multi-Model Chinese Correctors - Combines rule-based, statistical, and deep learning models to correct Chinese text errors.
• Rule-Based Chinese Correctors - Detects misspelled characters via language model perplexity and corrects them using phonetic and shape-similar features.
• Custom Model Training - Supports fine-tuning pretrained language models on user-annotated datasets to adapt error correction to specific domains.
• Error Correction Model Training - Supports fine-tuning pretrained language models on labeled Chinese text pairs to produce custom error correctors.
• T5 Correction Model Predictions - Provides a trained T5 model that corrects Chinese text and annotates errors in the output.
• Pretrained Model Loading - Loads pretrained correction model checkpoints from HuggingFace for Chinese text error correction without retraining.
• Custom Language Model Correctors - Loads a user-supplied Kenlm language model to tailor spelling correction to a specific domain.
• Error Correction Model Training - Provides training scripts for BERT models on paired error-correction data to learn character-level corrections.
• N-Gram Frequency Rankers - Ranks candidate corrections by counting character n-gram frequencies in a corpus without requiring word segmentation.
• Batch Text Correctors - Provides a command-line interface for batch processing Chinese text files with configurable correction options.
• Automated Text Corrections - Runs batch text correction on a file from the terminal using a language model, with options to control output detail and character detection.
• Batch Text Correctors - Runs batch Chinese text correction on a file from the terminal with configurable output and detail level.
• Chinese Grammar Error Corrections - Corrects Chinese grammar errors like missing or extra characters using sequence-to-sequence models.
• GPT-Based Chinese Correctors - Ships a GPT-based corrector that detects and fixes phonetic, shape-similar, and grammatical errors in Chinese sentences.
• N-Gram Language Model Corrections - Scores candidate corrections with a Kenlm n-gram language model to detect and fix ungrammatical sequences.
• Pinyin-to-Text Converters - Converts pinyin input to the most likely Chinese character sequence using a statistical language model and n-gram frequency counts.
• Correction Accuracy Evaluators - Provides evaluation tools to measure precision, recall, and F1 scores of correction models on benchmark test sets.
• AI - Listed in the “AI 项目” section of the Great Open Source Project awesome list.

Star history