Awesome GitHub RepositoriesParallel Text Processing

xi-editor is a high-performance text editor core written in Rust. It employs a client-server architecture that separates the backend editor logic from the user interface, allowing diverse frontends to communicate with the core via a standardized protocol. The project is distinguished by its use of rope-based text buffers for efficient manipulation of large documents and a collaborative engine powered by conflict-free replicated data types to synchronize concurrent edits. It further features an extensible plugin system that integrates external binaries and third-party tools through JSON-based

Compromise is a natural language processing library and rule-based text parser designed to analyze unstructured text. It functions as a toolkit for identifying parts of speech, linguistic patterns, and semantic meaning, while providing specialized engines for named entity recognition and the parsing of temporal and numeric data. The project is distinguished by its linguistic morphological engine, which can conjugate verbs across different tenses and inflect nouns and adjectives. It further allows for linguistic model customization through a plugin system that enables the extension of lexicons

This project is a high-performance library for converting raw text into tokens and IDs for machine learning models. It functions as a fast text encoder and a text preprocessing pipeline designed to transform strings into numerical representations with high throughput for research and production. The library includes a subword tokenizer trainer used to analyze text datasets and create custom vocabularies using algorithms such as byte-pair encoding and wordpiece. It provides capabilities for subword vocabulary training and text alignment, allowing character offsets to be tracked during normaliz

pkuseg-python is a Chinese word segmentation toolkit and natural language processing library. It provides specialized models for splitting Chinese text into words across various domains, including news, medical, and web content, and includes a tool for assigning grammatical parts of speech tags to segmented words. The library allows for the training of custom segmentation models using annotated datasets and supports the integration of user-defined dictionaries to ensure specialized terminology is recognized correctly. It employs a multi-threaded execution engine to process large volumes of Ch

LAC 是一个中文词法分析引擎和工具包，专为联合分词、词性标注和命名实体识别而设计。它作为一个高性能系统，利用训练好的机器学习模型来识别词边界和语法类别。 该项目具有轻量级的编译原生运行时，支持端侧自然语言处理并嵌入到移动应用程序中。它包含模型压缩和转换功能，以针对资源受限的环境进行优化，并支持多线程并行执行以提高吞吐量。 该工具包提供了联合词法分析功能，包括为词重要性评分分配数值权重。用户可以通过自定义词典覆盖来优化预测，或通过在特定领域数据集上进行增量模型训练来提高特定行业的准确性。