Awesome GitHub RepositoriesParallel Text Processing

xi-editor is a high-performance text editor core written in Rust. It employs a client-server architecture that separates the backend editor logic from the user interface, allowing diverse frontends to communicate with the core via a standardized protocol. The project is distinguished by its use of rope-based text buffers for efficient manipulation of large documents and a collaborative engine powered by conflict-free replicated data types to synchronize concurrent edits. It further features an extensible plugin system that integrates external binaries and third-party tools through JSON-based

Compromise is a natural language processing library and rule-based text parser designed to analyze unstructured text. It functions as a toolkit for identifying parts of speech, linguistic patterns, and semantic meaning, while providing specialized engines for named entity recognition and the parsing of temporal and numeric data. The project is distinguished by its linguistic morphological engine, which can conjugate verbs across different tenses and inflect nouns and adjectives. It further allows for linguistic model customization through a plugin system that enables the extension of lexicons

This project is a high-performance library for converting raw text into tokens and IDs for machine learning models. It functions as a fast text encoder and a text preprocessing pipeline designed to transform strings into numerical representations with high throughput for research and production. The library includes a subword tokenizer trainer used to analyze text datasets and create custom vocabularies using algorithms such as byte-pair encoding and wordpiece. It provides capabilities for subword vocabulary training and text alignment, allowing character offsets to be tracked during normaliz

pkuseg-python is a Chinese word segmentation toolkit and natural language processing library. It provides specialized models for splitting Chinese text into words across various domains, including news, medical, and web content, and includes a tool for assigning grammatical parts of speech tags to segmented words. The library allows for the training of custom segmentation models using annotated datasets and supports the integration of user-defined dictionaries to ensure specialized terminology is recognized correctly. It employs a multi-threaded execution engine to process large volumes of Ch

LAC هو محرك ومجموعة أدوات للتحليل المعجمي للغة الصينية مصمم للتقسيم المشترك للكلمات، ووضع علامات على أجزاء الكلام، والتعرف على الكيانات المسماة. يعمل كنظام عالي الأداء يحدد حدود الكلمات والفئات النحوية باستخدام نماذج تعلم آلة مدربة. يتميز المشروع بوقت تشغيل أصلي مجمع وخفيف الوزن يتيح معالجة اللغة الطبيعية على الجهاز والتضمين في تطبيقات الهاتف المحمول. يتضمن ضغط النماذج وتحويلها للتحسين للبيئات محدودة الموارد ويدعم التنفيذ المتوازي متعدد الخيوط لزيادة الإنتاجية. توفر مجموعة الأدوات قدرات للتحليل المعجمي المشترك، بما في ذلك القدرة على تعيين أوزان رقمية لتسجيل أهمية الكلمات. يمكن للمستخدمين تحسين التنبؤات من خلال تجاوزات القاموس المخصصة أو تحسين الدقة لصناعات محددة باستخدام تدريب النموذج التزايدي على مجموعات بيانات خاصة بالمجال.