Awesome GitHub RepositoriesParallel Text Processing

xi-editor is a high-performance text editor core written in Rust. It employs a client-server architecture that separates the backend editor logic from the user interface, allowing diverse frontends to communicate with the core via a standardized protocol. The project is distinguished by its use of rope-based text buffers for efficient manipulation of large documents and a collaborative engine powered by conflict-free replicated data types to synchronize concurrent edits. It further features an extensible plugin system that integrates external binaries and third-party tools through JSON-based

Compromise is a natural language processing library and rule-based text parser designed to analyze unstructured text. It functions as a toolkit for identifying parts of speech, linguistic patterns, and semantic meaning, while providing specialized engines for named entity recognition and the parsing of temporal and numeric data. The project is distinguished by its linguistic morphological engine, which can conjugate verbs across different tenses and inflect nouns and adjectives. It further allows for linguistic model customization through a plugin system that enables the extension of lexicons

This project is a high-performance library for converting raw text into tokens and IDs for machine learning models. It functions as a fast text encoder and a text preprocessing pipeline designed to transform strings into numerical representations with high throughput for research and production. The library includes a subword tokenizer trainer used to analyze text datasets and create custom vocabularies using algorithms such as byte-pair encoding and wordpiece. It provides capabilities for subword vocabulary training and text alignment, allowing character offsets to be tracked during normaliz

pkuseg-python is a Chinese word segmentation toolkit and natural language processing library. It provides specialized models for splitting Chinese text into words across various domains, including news, medical, and web content, and includes a tool for assigning grammatical parts of speech tags to segmented words. The library allows for the training of custom segmentation models using annotated datasets and supports the integration of user-defined dictionaries to ensure specialized terminology is recognized correctly. It employs a multi-threaded execution engine to process large volumes of Ch

LAC ist eine chinesische lexikalische Analyse-Engine und ein Toolkit, das für die gemeinsame Wortsegmentierung, Part-of-Speech-Tagging und Named-Entity-Recognition entwickelt wurde. Es fungiert als hochperformantes System, das Wortgrenzen und grammatikalische Kategorien mithilfe trainierter Machine-Learning-Modelle identifiziert. Das Projekt bietet eine leichtgewichtige, kompilierte native Runtime, die Natural Language Processing (NLP) auf dem Gerät und die Einbettung in mobile Anwendungen ermöglicht. Es enthält Modellkomprimierung und -konvertierung zur Optimierung für ressourcenbeschränkte Umgebungen und unterstützt Multi-Threaded-Parallel-Ausführung zur Steigerung des Durchsatzes. Das Toolkit bietet Funktionen für die gemeinsame lexikalische Analyse, einschließlich der Möglichkeit, numerische Gewichte für die Wichtigkeit von Wörtern zuzuweisen. Benutzer können Vorhersagen durch benutzerdefinierte Wörterbuch-Overrides verfeinern oder die Genauigkeit für spezifische Branchen durch inkrementelles Modelltraining auf domänenspezifischen Datensätzen verbessern.