Awesome GitHub RepositoriesMultilingual Text Processing

Pretext is a canvas-based text layout engine designed to calculate precise text dimensions and line breaks for custom rendering. It serves as a rich text measurement tool and a cross-browser typography normalizer, enabling the determination of pixel-perfect widths and heights for mixed inline content without relying on browser CSS. The project distinguishes itself through its ability to handle complex typography and dynamic layouts. It implements language-specific segmentation rules for CJK and Hangul scripts and corrects emoji width variances between DOM and canvas rendering. Additionally, i

HanLP is a natural language processing library and deep learning framework specifically optimized for the Chinese language, while also functioning as a multilingual text processor. It serves as a toolkit for performing linguistic analysis, semantic understanding, and script conversion. The project distinguishes itself through a dedicated focus on Chinese linguistic structures, including a specialized script converter for transforming text between Simplified Chinese, Traditional Chinese, and Pinyin. It further supports domain-specific model training to improve the recognition of professional t

Fairseq is a PyTorch toolkit for sequence-to-sequence modeling, specializing in neural machine translation, automatic speech recognition, and large-scale language model training. It provides a framework for processing and aligning diverse data sources, including text, audio, and video, to support tasks such as speech-to-text conversion and multimodal sequence learning. The project is distinguished by its distributed training capabilities, which utilize parameter sharding, mixed-precision training, and CPU offloading to handle models that exceed single-device memory. It also includes specializ

EasyOCR is a deep learning-based computer vision library designed to perform optical character recognition on images and video frames. It functions as a comprehensive pipeline that automates the transformation of visual text into machine-readable strings, enabling the digitization of physical documents, forms, and receipts into searchable data. The engine distinguishes itself through a multi-stage processing workflow that combines convolutional neural networks for spatial feature extraction with sequence-based decoding mechanisms. This architecture allows the system to identify and interpret

Qwen2.5 is a suite of large language model foundation models designed for natural language generation, code production, and complex mathematical reasoning. The project encompasses a multilingual language model capable of processing dozens of languages and a specialized code generation model for technical problem solving and debugging. The framework is distinguished by its long context capabilities, enabling the analysis of massive inputs ranging from 256K up to 1 million tokens. It further functions as an agentic framework, utilizing standardized templates and parsers to execute autonomous wo

openai-translator is a cross-platform translation tool and language learning utility available as a browser extension and desktop application. It functions as a client for large language model APIs to translate, summarize, and polish text across different digital environments. The project differentiates itself by integrating optical character recognition to translate text extracted from images and screenshots. It also includes a language learning workflow that allows users to save new vocabulary to a digital book and use text-to-speech synthesis for pronunciation. The tool provides broad tex

This project is a comprehensive framework and toolkit for developing, optimizing, and deploying transformer-based models across multimodal, document intelligence, and natural language processing tasks. It provides a unified neural architecture that processes text, vision, audio, and document layout data through a shared set of weights, enabling researchers and developers to build foundational models that align cross-modal representations. The platform distinguishes itself through advanced training and inference strategies designed for large-scale deep learning. It incorporates specialized mec

Sonic is a high-performance, lightweight search backend designed to provide real-time full-text search and autocomplete capabilities for applications. It functions as a persistent indexing server that maps text terms to object identifiers, allowing developers to integrate rapid search functionality without storing raw document content directly within the search engine. The system distinguishes itself through a specialized graph-based index that enables real-time word prediction and typo correction. Communication is handled via a custom, low-latency binary protocol over raw TCP sockets, which

Qwen2-VL is a multimodal large language model and vision language model designed to process and reason across text, images, and video content. It functions as a visual reasoning engine and a visual agent framework, capable of interpreting visual data to perform object detection, document parsing, and spatial reasoning. The model is distinguished by its ability to act as a video understanding model, processing hour-long videos with second-level indexing and event recall. It further differentiates itself through a visual agent capability that interacts with software interfaces and robotic hardw

This project is a comprehensive, curated directory of static analysis, linting, and security scanning utilities. It serves as a central resource for developers to discover, compare, and select tools based on specific programming languages, licensing models, and integration requirements. The directory distinguishes itself by providing deep metadata for each listed utility, including community-driven popularity rankings, maintenance status, and deployment methods. By aggregating these tools into a single searchable index, it enables teams to identify solutions for enforcing coding standards, ma

Maptoposter is a geographic data visualization engine that transforms raw OpenStreetMap data into high-quality, minimalist map posters. It functions as a generator that converts complex urban layouts and transportation networks into stylized visual representations suitable for large-scale physical printing. The tool distinguishes itself through a specialized rendering pipeline that combines vector-based graphic generation with a script-aware typography engine. This allows for the automatic detection of global writing systems, ensuring that map labels are correctly formatted and spaced regardl

imewlconverter is an input method editor wordlist converter and format transformer designed to migrate user dictionaries and phrase lists between different software environments. It functions as a cross-platform dictionary migrator, translating proprietary binary and text wordlists for use across Windows, macOS, and mobile systems. The tool standardizes diverse lexicon formats, such as WL, FIT, DCTX, LD2, and QPYD, into common structures to ensure cross-platform compatibility. It specifically handles binary wordlist extraction and the transformation of custom phrase lists for systems includin

Dango-Translator is an OCR translation system and multi-engine translation client designed to extract text from images or screens and replace it with translated content. It functions as an image text translator and real-time screen translator, utilizing optical character recognition to convert text between different languages automatically. The software distinguishes itself through coordinate-based image typesetting and a glossary manager. These tools allow for the replacement of original image content with translated text in the same area and the use of specialized dictionaries to ensure con

Yi is a bilingual language model and foundation model designed for natural language processing, reasoning, and reading comprehension in both English and Chinese. It is built as a transformer-based architecture capable of general purpose text generation and conversational tasks. The model is distinguished by its ability to function as a long context system, processing and analyzing extended input sequences up to 200k tokens. It also supports quantized versions that use low-bit precision to reduce memory footprints, enabling execution on consumer-grade hardware. The project covers a broad rang

This repository provides the pre-trained neural network and legacy data files used by Tesseract to recognize and extract printed text from images. It serves as a multilingual training data repository and a collection of Long Short-Term Memory models designed for high-accuracy optical character recognition across various global scripts and languages. The data includes specialized models for analyzing image layouts to determine text rotation and script direction. It provides the necessary language-specific datasets and linguistic patterns required to enable Tesseract OCR engines to function. T

GLM-4 is a large language model and fine-tuning framework designed for human-like text production, complex reasoning, and multilingual conversation. It functions as a multimodal system capable of processing high-resolution visual content and as a long-context model designed to analyze documents with a context window of up to one million tokens. The project differentiates itself through a function calling interface that enables AI agent development by connecting the model to external APIs and real-time web browsing. It includes specialized capabilities for generating functional programming cod

pdfminer.six is a programmatic tool for extracting text, layout information, and metadata from PDF documents into machine-readable formats. It functions as a document parser that converts internal PDF objects and structures into accessible data objects for analysis. The project includes utilities for decrypting RC4 and AES encrypted files to enable content extraction. It also provides a layout analyzer to identify fonts, colors, and text locations to determine the organizational structure of pages. The system covers a broad range of extraction capabilities, including the retrieval of embedde

Franc ist eine Bibliothek zur Erkennung natürlicher Sprachen und ein Befehlszeilen-Identifikator, der verwendet wird, um die geschriebene Sprache einer Textprobe zu bestimmen. Es fungiert als statistischer Sprachprofiler, der mehrsprachigen Text durch die Analyse von Zeichenverteilungen identifiziert und klassifiziert. Das Tool verwendet ein Trigram-basiertes statistisches Analysesystem, das die Häufigkeit von Drei-Zeichen-Sequenzen in einer Eingabeprobe mit Referenzprofilen vergleicht. Es ordnet potenzielle Sprachübereinstimmungen durch die Berechnung der statistischen Distanz zwischen der Eingabe und diesen Profilen und ermöglicht so die Rückgabe einer Rangliste wahrscheinlicher Sprachen. Das Projekt bietet eine Befehlszeilenschnittstelle für Textanalyse und unterstützt automatisierte Inhaltsklassifizierung. Es nutzt modulare Sprachdatensätze und vorberechnete N-Gramm-Tabellen, um die Erkennungslogik von sprachspezifischen Profildaten zu trennen.

Chinese-Vicuna ist ein chinesisches Large Language Model und eine instruction-following KI auf Basis der LLaMA-Architektur. Es wurde speziell für das Verständnis und die Generierung natürlicher Sprache in Chinesisch entwickelt und nutzt ein instruction-tuned Modell, um komplexen Benutzeranweisungen in Konversationen zu folgen. Das Projekt bietet ein LoRA-Fine-Tuning-Framework und Quantisierungssysteme, um Modelladaption und Inferenz auf Consumer-Hardware zu ermöglichen. Es implementiert quantisierte Inferenz, um den Speicherverbrauch auf CPUs und GPUs zu reduzieren, unterstützt durch eine C++-Implementierung auf niedriger Ebene, um Systemressourcenanforderungen zu minimieren. Das System deckt ein breites Spektrum an NLP-Funktionen ab, einschließlich Multi-Turn-Konversationsmanagement, mehrsprachiger Übersetzung und Generierung von Programmiercode. Es enthält zudem Tools für domänenspezifisches Training, Modellformat-Konvertierung und ein interaktives Chat-Interface mit Streaming-Textausgabe.

python-goose is a Python library for web scraping and content extraction. It functions as an HTML boilerplate remover and article parser designed to isolate primary text and metadata from web pages by stripping away navigation, layout noise, and non-essential elements. The tool features multilingual processing capabilities, utilizing language-specific stop-word analyzers to identify and extract primary content across different languages. It also identifies and collects embedded media, including source URLs and embed codes for lead images and videos associated with an article. The library cov