baidu/lac
Lac
LAC هو محرك ومجموعة أدوات للتحليل المعجمي للغة الصينية مصمم للتقسيم المشترك للكلمات، ووضع علامات على أجزاء الكلام، والتعرف على الكيانات المسماة. يعمل كنظام عالي الأداء يحدد حدود الكلمات والفئات النحوية باستخدام نماذج تعلم آلة مدربة.
يتميز المشروع بوقت تشغيل أصلي مجمع وخفيف الوزن يتيح معالجة اللغة الطبيعية على الجهاز والتضمين في تطبيقات الهاتف المحمول. يتضمن ضغط النماذج وتحويلها للتحسين للبيئات محدودة الموارد ويدعم التنفيذ المتوازي متعدد الخيوط لزيادة الإنتاجية.
توفر مجموعة الأدوات قدرات للتحليل المعجمي المشترك، بما في ذلك القدرة على تعيين أوزان رقمية لتسجيل أهمية الكلمات. يمكن للمستخدمين تحسين التنبؤات من خلال تجاوزات القاموس المخصصة أو تحسين الدقة لصناعات محددة باستخدام تدريب النموذج التزايدي على مجموعات بيانات خاصة بالمجال.
Features
- Chinese Natural Language Processing - Provides a comprehensive suite for Chinese natural language processing including segmentation, tagging, and entity recognition.
- Joint Lexical Analysis - Provides a high-performance engine for joint word segmentation, part-of-speech tagging, and named entity recognition in Chinese text.
- Chinese NLP Libraries - Ships a toolkit specifically designed for the joint linguistic analysis of Chinese text.
- Lexical Analysis Engines - Implements a high-performance engine that identifies word boundaries and grammatical categories using trained models.
- Model Inference Engines - Implements a high-performance inference engine that performs joint lexical analysis in a single pass.
- Named Entity Recognition - Includes a processor that categorizes specific entities like people, locations, and organizations within text.
- Native Runtime Libraries - Ships a compiled native runtime library for high-performance model execution on mobile and desktop devices.
- Local NLP Inference - Enables local execution of NLP tasks on-device to operate without network connectivity.
- Part-of-Speech Taggers - Assigns grammatical categories and named entity labels to segmented Chinese words using machine learning models.
- Chinese Language Segmenters - Provides specialized tools for tokenizing and segmenting continuous Chinese text streams.
- Mobile Framework Integrations - Offers lightweight runtime libraries for embedding NLP models directly into mobile applications.
- Incremental Model Training - Supports incremental training on custom datasets to adapt the model to specific industry domains.
- Model Compression - Includes model compression and conversion to optimize computation graphs for resource-constrained environments.
- Keyword Importance Scoring - Assigns numerical weights to words to distinguish core meaning from redundant terms in a query.
- Domain-Specific Fine-Tuning - Supports improving lexical analysis accuracy through incremental fine-tuning on domain-specific datasets.
- Lexicon-Driven Segmenters - Combines machine learning models with user-defined lexicons to refine text segmentation and tagging.
- Custom Dictionary Managers - Provides custom dictionary managers to refine model predictions for word boundaries and labels.
- Prediction Overrides - Allows forcing specific word boundaries and labels using user-defined custom lexicons to override model predictions.
- NLP Mobile Runtimes - Provides a lightweight, compiled interface for executing linguistic analysis models locally on mobile devices.
- Model Compression - Includes model compression and conversion to optimize performance for resource-constrained mobile environments.
- C++ Inference Runtimes - Provides a compiled C++ runtime optimized for executing machine learning model inference.
- Parallel Text Processing - Supports multi-threaded parallel execution to distribute text processing tasks across CPU cores.
- Chinese NLP Toolkits - Baidu's open-source tool for segmentation, tagging, and entity recognition.
