Awesome GitHub RepositoriesMultilingual Text Processing

Pretext is a canvas-based text layout engine designed to calculate precise text dimensions and line breaks for custom rendering. It serves as a rich text measurement tool and a cross-browser typography normalizer, enabling the determination of pixel-perfect widths and heights for mixed inline content without relying on browser CSS. The project distinguishes itself through its ability to handle complex typography and dynamic layouts. It implements language-specific segmentation rules for CJK and Hangul scripts and corrects emoji width variances between DOM and canvas rendering. Additionally, i

HanLP is a natural language processing library and deep learning framework specifically optimized for the Chinese language, while also functioning as a multilingual text processor. It serves as a toolkit for performing linguistic analysis, semantic understanding, and script conversion. The project distinguishes itself through a dedicated focus on Chinese linguistic structures, including a specialized script converter for transforming text between Simplified Chinese, Traditional Chinese, and Pinyin. It further supports domain-specific model training to improve the recognition of professional t

Fairseq is a PyTorch toolkit for sequence-to-sequence modeling, specializing in neural machine translation, automatic speech recognition, and large-scale language model training. It provides a framework for processing and aligning diverse data sources, including text, audio, and video, to support tasks such as speech-to-text conversion and multimodal sequence learning. The project is distinguished by its distributed training capabilities, which utilize parameter sharding, mixed-precision training, and CPU offloading to handle models that exceed single-device memory. It also includes specializ

EasyOCR is a deep learning-based computer vision library designed to perform optical character recognition on images and video frames. It functions as a comprehensive pipeline that automates the transformation of visual text into machine-readable strings, enabling the digitization of physical documents, forms, and receipts into searchable data. The engine distinguishes itself through a multi-stage processing workflow that combines convolutional neural networks for spatial feature extraction with sequence-based decoding mechanisms. This architecture allows the system to identify and interpret

Qwen2.5 is a suite of large language model foundation models designed for natural language generation, code production, and complex mathematical reasoning. The project encompasses a multilingual language model capable of processing dozens of languages and a specialized code generation model for technical problem solving and debugging. The framework is distinguished by its long context capabilities, enabling the analysis of massive inputs ranging from 256K up to 1 million tokens. It further functions as an agentic framework, utilizing standardized templates and parsers to execute autonomous wo

openai-translator is a cross-platform translation tool and language learning utility available as a browser extension and desktop application. It functions as a client for large language model APIs to translate, summarize, and polish text across different digital environments. The project differentiates itself by integrating optical character recognition to translate text extracted from images and screenshots. It also includes a language learning workflow that allows users to save new vocabulary to a digital book and use text-to-speech synthesis for pronunciation. The tool provides broad tex

This project is a comprehensive framework and toolkit for developing, optimizing, and deploying transformer-based models across multimodal, document intelligence, and natural language processing tasks. It provides a unified neural architecture that processes text, vision, audio, and document layout data through a shared set of weights, enabling researchers and developers to build foundational models that align cross-modal representations. The platform distinguishes itself through advanced training and inference strategies designed for large-scale deep learning. It incorporates specialized mec

Sonic is a high-performance, lightweight search backend designed to provide real-time full-text search and autocomplete capabilities for applications. It functions as a persistent indexing server that maps text terms to object identifiers, allowing developers to integrate rapid search functionality without storing raw document content directly within the search engine. The system distinguishes itself through a specialized graph-based index that enables real-time word prediction and typo correction. Communication is handled via a custom, low-latency binary protocol over raw TCP sockets, which

Qwen2-VL is a multimodal large language model and vision language model designed to process and reason across text, images, and video content. It functions as a visual reasoning engine and a visual agent framework, capable of interpreting visual data to perform object detection, document parsing, and spatial reasoning. The model is distinguished by its ability to act as a video understanding model, processing hour-long videos with second-level indexing and event recall. It further differentiates itself through a visual agent capability that interacts with software interfaces and robotic hardw

This project is a comprehensive, curated directory of static analysis, linting, and security scanning utilities. It serves as a central resource for developers to discover, compare, and select tools based on specific programming languages, licensing models, and integration requirements. The directory distinguishes itself by providing deep metadata for each listed utility, including community-driven popularity rankings, maintenance status, and deployment methods. By aggregating these tools into a single searchable index, it enables teams to identify solutions for enforcing coding standards, ma

Maptoposter is a geographic data visualization engine that transforms raw OpenStreetMap data into high-quality, minimalist map posters. It functions as a generator that converts complex urban layouts and transportation networks into stylized visual representations suitable for large-scale physical printing. The tool distinguishes itself through a specialized rendering pipeline that combines vector-based graphic generation with a script-aware typography engine. This allows for the automatic detection of global writing systems, ensuring that map labels are correctly formatted and spaced regardl

imewlconverter is an input method editor wordlist converter and format transformer designed to migrate user dictionaries and phrase lists between different software environments. It functions as a cross-platform dictionary migrator, translating proprietary binary and text wordlists for use across Windows, macOS, and mobile systems. The tool standardizes diverse lexicon formats, such as WL, FIT, DCTX, LD2, and QPYD, into common structures to ensure cross-platform compatibility. It specifically handles binary wordlist extraction and the transformation of custom phrase lists for systems includin

Dango-Translator is an OCR translation system and multi-engine translation client designed to extract text from images or screens and replace it with translated content. It functions as an image text translator and real-time screen translator, utilizing optical character recognition to convert text between different languages automatically. The software distinguishes itself through coordinate-based image typesetting and a glossary manager. These tools allow for the replacement of original image content with translated text in the same area and the use of specialized dictionaries to ensure con

Yi is a bilingual language model and foundation model designed for natural language processing, reasoning, and reading comprehension in both English and Chinese. It is built as a transformer-based architecture capable of general purpose text generation and conversational tasks. The model is distinguished by its ability to function as a long context system, processing and analyzing extended input sequences up to 200k tokens. It also supports quantized versions that use low-bit precision to reduce memory footprints, enabling execution on consumer-grade hardware. The project covers a broad rang

This repository provides the pre-trained neural network and legacy data files used by Tesseract to recognize and extract printed text from images. It serves as a multilingual training data repository and a collection of Long Short-Term Memory models designed for high-accuracy optical character recognition across various global scripts and languages. The data includes specialized models for analyzing image layouts to determine text rotation and script direction. It provides the necessary language-specific datasets and linguistic patterns required to enable Tesseract OCR engines to function. T

GLM-4 is a large language model and fine-tuning framework designed for human-like text production, complex reasoning, and multilingual conversation. It functions as a multimodal system capable of processing high-resolution visual content and as a long-context model designed to analyze documents with a context window of up to one million tokens. The project differentiates itself through a function calling interface that enables AI agent development by connecting the model to external APIs and real-time web browsing. It includes specialized capabilities for generating functional programming cod

pdfminer.six is a programmatic tool for extracting text, layout information, and metadata from PDF documents into machine-readable formats. It functions as a document parser that converts internal PDF objects and structures into accessible data objects for analysis. The project includes utilities for decrypting RC4 and AES encrypted files to enable content extraction. It also provides a layout analyzer to identify fonts, colors, and text locations to determine the organizational structure of pages. The system covers a broad range of extraction capabilities, including the retrieval of embedde

Franc este o bibliotecă de detecție a limbajului natural și un identificator în linia de comandă utilizat pentru a determina limba scrisă a unui eșantion de text. Acesta funcționează ca un profiler statistic de limbaj care identifică și clasifică textul multilingv prin analizarea distribuțiilor de caractere. Instrumentul utilizează un sistem de analiză statistică bazat pe trigrame, comparând frecvența secvențelor de trei caractere dintr-un eșantion de intrare cu profilurile de referință. Acesta clasează potențialele potriviri de limbă prin calcularea distanței statistice dintre input și aceste profiluri, permițându-i să returneze o listă clasată a limbilor probabile. Proiectul oferă o interfață în linia de comandă pentru analiza textului și suportă clasificarea automată a conținutului. Utilizează seturi de date lingvistice modulare și tabele n-gram precalculate pentru a separa logica de detecție de datele de profil specifice limbii.

Chinese-Vicuna este un model de limbaj mare (LLM) chinezesc și o AI care urmează instrucțiuni, bazată pe arhitectura LLaMA. Este concepută special pentru înțelegerea și generarea limbajului natural în limba chineză, utilizând un model ajustat prin instrucțiuni pentru a urma prompt-urile complexe ale utilizatorilor în timpul conversațiilor. Proiectul oferă un framework de fine-tuning LoRA și sisteme de cuantizare pentru a permite adaptarea modelului și inferența pe hardware de consum. Implementează inferența cuantizată pentru a reduce utilizarea memoriei atât pe CPU, cât și pe GPU, susținută de o implementare C++ de nivel scăzut pentru a minimiza cerințele de resurse ale sistemului. Sistemul acoperă o gamă largă de capabilități de procesare a limbajului natural, inclusiv gestionarea conversațiilor multi-turn, traducerea multilingvă și generarea de cod de programare. Include, de asemenea, instrumente pentru antrenarea specifică domeniului, conversia formatului modelului și o interfață de chat interactivă cu output de text în streaming.

python-goose este o bibliotecă Python pentru web scraping și extragerea conținutului. Aceasta funcționează ca un eliminator de boilerplate HTML și un parser de articole conceput pentru a izola textul principal și metadatele din paginile web prin eliminarea navigării, a zgomotului de layout și a elementelor neesențiale. Instrumentul dispune de capabilități de procesare multilingvă, utilizând analizoare de stop-words specifice limbii pentru a identifica și extrage conținutul principal în diferite limbi. De asemenea, identifică și colectează conținutul media încorporat, inclusiv URL-urile sursă și codurile de embed pentru imaginile principale și videoclipurile asociate cu un articol. Biblioteca acoperă domenii funcționale largi, inclusiv web scraping automatizat, extragerea conținutului structurat și colectarea țintită a conținutului media din articole.