Awesome GitHub RepositoriesText Processing

re2 is a C++ regular expression library designed for high-performance text processing. It is a non-backtracking regex engine that provides linear-time pattern matching, ensuring that execution time remains proportional to the size of the input string regardless of the pattern used. The library supports UTF-8 and Latin-1 text encodings for searching and extracting substrings. It includes capabilities for multi-pattern optimization, allowing multiple regular expressions to be combined into a single representation to scan text for several patterns in one pass. The project covers core regex oper

pkuseg-python is a Chinese word segmentation toolkit and natural language processing library. It provides specialized models for splitting Chinese text into words across various domains, including news, medical, and web content, and includes a tool for assigning grammatical parts of speech tags to segmented words. The library allows for the training of custom segmentation models using annotated datasets and supports the integration of user-defined dictionaries to ensure specialized terminology is recognized correctly. It employs a multi-threaded execution engine to process large volumes of Ch

This project is a Python port of the libphonenumber library, providing a suite of tools for parsing, formatting, and validating international phone numbers. It converts raw phone number strings into structured objects based on international numbering plans and verifies if numbers are possible or valid according to digit length and assigned exchange prefixes. The library provides metadata services to retrieve the original network carrier, geographical location descriptions, and associated time zones for specific phone numbers. It also includes tools for transforming numbers into standardized i

Python character encoding detector

Rhubarb is an automated lip sync generator and phonetic speech analyzer that converts audio recordings into timed mouth-shape animation data. It identifies sounds and syllables within audio files to map them to specific visual mouth shapes, serving as an animation timing exporter for external character animation software. The tool utilizes a language-independent phonetic recognizer to process speech regardless of the spoken language. To increase accuracy, it supports dialogue-guided recognition by using external text files to guide the phonetic analysis of specific spoken scripts. The system

emoji terminal output for Python

python humanize functions

This is a set of command line utilities for manipulating large tabular data files. Files of numeric and text data commonly found in machine learning and data mining environments. Filtering, sampling, statistics, joins, and more.

Python module (C extension and plain python) implementing Aho-Corasick algorithm

A markdown parsing and rendering library for D programming language. The project use commonmark spec, ported from commonmark-java.