5 रिपॉजिटरी
Awesome GitHub RepositoriesParallel Text Processing
Distributing linguistic parsing tasks across multiple CPU cores to increase throughput.
Distinct from Text Processing and Parsing: Candidates focused on visual parallel reading or AST caching, not multi-threaded parsing execution.
Explore 5 awesome GitHub repositories matching software engineering & architecture · Parallel Text Processing. Refine with filters or upvote what's useful.
Awesome Parallel Text Processing GitHub Repositories
AI के साथ बेहतरीन रिपॉजिटरी खोजें।हम AI का उपयोग करके सबसे सटीक रिपॉजिटरी खोजेंगे।
- xi-editor/xi-editor
xi-editor/xi-editor
19,816GitHub पर देखेंxi-editor is a high-performance text editor core written in Rust. It employs a client-server architecture that separates the backend editor logic from the user interface, allowing diverse frontends to communicate with the core via a standardized protocol. The project is distinguished by its use of rope-based text buffers for efficient manipulation of large documents and a collaborative engine powered by conflict-free replicated data types to synchronize concurrent edits. It further features an extensible plugin system that integrates external binaries and third-party tools through JSON-based
Distributes expensive text operations like word wrapping across multiple CPU cores using a MapReduce approach.
Rust - spencermountain/compromise
spencermountain/compromise
12,125GitHub पर देखेंCompromise is a natural language processing library and rule-based text parser designed to analyze unstructured text. It functions as a toolkit for identifying parts of speech, linguistic patterns, and semantic meaning, while providing specialized engines for named entity recognition and the parsing of temporal and numeric data. The project is distinguished by its linguistic morphological engine, which can conjugate verbs across different tenses and inflect nouns and adjectives. It further allows for linguistic model customization through a plugin system that enables the extension of lexicons
Distributes sentence parsing across multiple worker threads to process large volumes of text efficiently.
JavaScriptnamed-entity-recognitionnlppart-of-speech - huggingface/tokenizers
huggingface/tokenizers
10,825GitHub पर देखेंThis project is a high-performance library for converting raw text into tokens and IDs for machine learning models. It functions as a fast text encoder and a text preprocessing pipeline designed to transform strings into numerical representations with high throughput for research and production. The library includes a subword tokenizer trainer used to analyze text datasets and create custom vocabularies using algorithms such as byte-pair encoding and wordpiece. It provides capabilities for subword vocabulary training and text alignment, allowing character offsets to be tracked during normaliz
Implements multi-threaded parallel processing of text to increase encoding throughput across multiple CPU cores.
Rustbertgptlanguage-model - lancopku/pkuseg-python
lancopku/pkuseg-python
6,707GitHub पर देखेंpkuseg-python is a Chinese word segmentation toolkit and natural language processing library. It provides specialized models for splitting Chinese text into words across various domains, including news, medical, and web content, and includes a tool for assigning grammatical parts of speech tags to segmented words. The library allows for the training of custom segmentation models using annotated datasets and supports the integration of user-defined dictionaries to ensure specialized terminology is recognized correctly. It employs a multi-threaded execution engine to process large volumes of Ch
Distributes word segmentation tasks across multiple CPU threads to increase processing throughput.
Python - baidu/lac
baidu/lac
4,001GitHub पर देखेंLAC एक चीनी लेक्सिकल विश्लेषण इंजन और टूलकिट है जिसे संयुक्त शब्द विभाजन, पार्ट-ऑफ-स्पीच टैगिंग और नामित इकाई पहचान के लिए डिज़ाइन किया गया है। यह एक उच्च-प्रदर्शन सिस्टम के रूप में कार्य करता है जो प्रशिक्षित मशीन लर्निंग मॉडल का उपयोग करके शब्द सीमाओं और व्याकरणिक श्रेणियों की पहचान करता है। इस प्रोजेक्ट में एक हल्का, संकलित नेटिव रनटाइम है जो ऑन-डिवाइस नेचुरल लैंग्वेज प्रोसेसिंग और मोबाइल एप्लिकेशन में एम्बेडिंग को सक्षम बनाता है। इसमें संसाधन-विवश वातावरण के लिए अनुकूलित करने के लिए मॉडल संपीड़न और रूपांतरण शामिल है और थ्रूपुट बढ़ाने के लिए मल्टी-थ्रेडेड समानांतर निष्पादन का समर्थन करता है। यह टूलकिट संयुक्त लेक्सिकल विश्लेषण के लिए क्षमताएं प्रदान करती है, जिसमें शब्द महत्व स्कोरिंग के लिए संख्यात्मक भार निर्दिष्ट करने की क्षमता शामिल है। उपयोगकर्ता कस्टम शब्दकोश ओवरराइड के माध्यम से भविष्यवाणियों को परिष्कृत कर सकते हैं या डोमेन-विशिष्ट डेटासेट पर वृद्धिशील मॉडल प्रशिक्षण का उपयोग करके विशिष्ट उद्योगों के लिए सटीकता में सुधार कर सकते हैं।
Supports multi-threaded parallel execution to distribute text processing tasks across CPU cores.
C++chinese-nlpchinese-word-segmentationjava
