Awesome GitHub RepositoriesMultilingual Text Processing

Pretext is a canvas-based text layout engine designed to calculate precise text dimensions and line breaks for custom rendering. It serves as a rich text measurement tool and a cross-browser typography normalizer, enabling the determination of pixel-perfect widths and heights for mixed inline content without relying on browser CSS. The project distinguishes itself through its ability to handle complex typography and dynamic layouts. It implements language-specific segmentation rules for CJK and Hangul scripts and corrects emoji width variances between DOM and canvas rendering. Additionally, i

HanLP is a natural language processing library and deep learning framework specifically optimized for the Chinese language, while also functioning as a multilingual text processor. It serves as a toolkit for performing linguistic analysis, semantic understanding, and script conversion. The project distinguishes itself through a dedicated focus on Chinese linguistic structures, including a specialized script converter for transforming text between Simplified Chinese, Traditional Chinese, and Pinyin. It further supports domain-specific model training to improve the recognition of professional t

Fairseq is a PyTorch toolkit for sequence-to-sequence modeling, specializing in neural machine translation, automatic speech recognition, and large-scale language model training. It provides a framework for processing and aligning diverse data sources, including text, audio, and video, to support tasks such as speech-to-text conversion and multimodal sequence learning. The project is distinguished by its distributed training capabilities, which utilize parameter sharding, mixed-precision training, and CPU offloading to handle models that exceed single-device memory. It also includes specializ

EasyOCR is a deep learning-based computer vision library designed to perform optical character recognition on images and video frames. It functions as a comprehensive pipeline that automates the transformation of visual text into machine-readable strings, enabling the digitization of physical documents, forms, and receipts into searchable data. The engine distinguishes itself through a multi-stage processing workflow that combines convolutional neural networks for spatial feature extraction with sequence-based decoding mechanisms. This architecture allows the system to identify and interpret

Qwen2.5 is a suite of large language model foundation models designed for natural language generation, code production, and complex mathematical reasoning. The project encompasses a multilingual language model capable of processing dozens of languages and a specialized code generation model for technical problem solving and debugging. The framework is distinguished by its long context capabilities, enabling the analysis of massive inputs ranging from 256K up to 1 million tokens. It further functions as an agentic framework, utilizing standardized templates and parsers to execute autonomous wo

openai-translator is a cross-platform translation tool and language learning utility available as a browser extension and desktop application. It functions as a client for large language model APIs to translate, summarize, and polish text across different digital environments. The project differentiates itself by integrating optical character recognition to translate text extracted from images and screenshots. It also includes a language learning workflow that allows users to save new vocabulary to a digital book and use text-to-speech synthesis for pronunciation. The tool provides broad tex

This project is a comprehensive framework and toolkit for developing, optimizing, and deploying transformer-based models across multimodal, document intelligence, and natural language processing tasks. It provides a unified neural architecture that processes text, vision, audio, and document layout data through a shared set of weights, enabling researchers and developers to build foundational models that align cross-modal representations. The platform distinguishes itself through advanced training and inference strategies designed for large-scale deep learning. It incorporates specialized mec

Sonic is a high-performance, lightweight search backend designed to provide real-time full-text search and autocomplete capabilities for applications. It functions as a persistent indexing server that maps text terms to object identifiers, allowing developers to integrate rapid search functionality without storing raw document content directly within the search engine. The system distinguishes itself through a specialized graph-based index that enables real-time word prediction and typo correction. Communication is handled via a custom, low-latency binary protocol over raw TCP sockets, which

Qwen2-VL is a multimodal large language model and vision language model designed to process and reason across text, images, and video content. It functions as a visual reasoning engine and a visual agent framework, capable of interpreting visual data to perform object detection, document parsing, and spatial reasoning. The model is distinguished by its ability to act as a video understanding model, processing hour-long videos with second-level indexing and event recall. It further differentiates itself through a visual agent capability that interacts with software interfaces and robotic hardw

This project is a comprehensive, curated directory of static analysis, linting, and security scanning utilities. It serves as a central resource for developers to discover, compare, and select tools based on specific programming languages, licensing models, and integration requirements. The directory distinguishes itself by providing deep metadata for each listed utility, including community-driven popularity rankings, maintenance status, and deployment methods. By aggregating these tools into a single searchable index, it enables teams to identify solutions for enforcing coding standards, ma

Maptoposter is a geographic data visualization engine that transforms raw OpenStreetMap data into high-quality, minimalist map posters. It functions as a generator that converts complex urban layouts and transportation networks into stylized visual representations suitable for large-scale physical printing. The tool distinguishes itself through a specialized rendering pipeline that combines vector-based graphic generation with a script-aware typography engine. This allows for the automatic detection of global writing systems, ensuring that map labels are correctly formatted and spaced regardl

imewlconverter is an input method editor wordlist converter and format transformer designed to migrate user dictionaries and phrase lists between different software environments. It functions as a cross-platform dictionary migrator, translating proprietary binary and text wordlists for use across Windows, macOS, and mobile systems. The tool standardizes diverse lexicon formats, such as WL, FIT, DCTX, LD2, and QPYD, into common structures to ensure cross-platform compatibility. It specifically handles binary wordlist extraction and the transformation of custom phrase lists for systems includin

Dango-Translator is an OCR translation system and multi-engine translation client designed to extract text from images or screens and replace it with translated content. It functions as an image text translator and real-time screen translator, utilizing optical character recognition to convert text between different languages automatically. The software distinguishes itself through coordinate-based image typesetting and a glossary manager. These tools allow for the replacement of original image content with translated text in the same area and the use of specialized dictionaries to ensure con

Yi is a bilingual language model and foundation model designed for natural language processing, reasoning, and reading comprehension in both English and Chinese. It is built as a transformer-based architecture capable of general purpose text generation and conversational tasks. The model is distinguished by its ability to function as a long context system, processing and analyzing extended input sequences up to 200k tokens. It also supports quantized versions that use low-bit precision to reduce memory footprints, enabling execution on consumer-grade hardware. The project covers a broad rang

This repository provides the pre-trained neural network and legacy data files used by Tesseract to recognize and extract printed text from images. It serves as a multilingual training data repository and a collection of Long Short-Term Memory models designed for high-accuracy optical character recognition across various global scripts and languages. The data includes specialized models for analyzing image layouts to determine text rotation and script direction. It provides the necessary language-specific datasets and linguistic patterns required to enable Tesseract OCR engines to function. T

GLM-4 is a large language model and fine-tuning framework designed for human-like text production, complex reasoning, and multilingual conversation. It functions as a multimodal system capable of processing high-resolution visual content and as a long-context model designed to analyze documents with a context window of up to one million tokens. The project differentiates itself through a function calling interface that enables AI agent development by connecting the model to external APIs and real-time web browsing. It includes specialized capabilities for generating functional programming cod

pdfminer.six is a programmatic tool for extracting text, layout information, and metadata from PDF documents into machine-readable formats. It functions as a document parser that converts internal PDF objects and structures into accessible data objects for analysis. The project includes utilities for decrypting RC4 and AES encrypted files to enable content extraction. It also provides a layout analyzer to identify fonts, colors, and text locations to determine the organizational structure of pages. The system covers a broad range of extraction capabilities, including the retrieval of embedde

Franc est une bibliothèque de détection de langage naturel et un identifiant en ligne de commande utilisé pour déterminer la langue écrite d'un échantillon de texte. Il fonctionne comme un profileur de langue statistique qui identifie et classifie le texte multilingue en analysant les distributions de caractères. L'outil emploie un système d'analyse statistique basé sur les trigrammes, comparant la fréquence des séquences de trois caractères dans un échantillon d'entrée par rapport à des profils de référence. Il classe les correspondances de langue potentielles en calculant la distance statistique entre l'entrée et ces profils, lui permettant de retourner une liste classée des langues probables. Le projet fournit une interface en ligne de commande pour l'analyse de texte et prend en charge la classification automatisée de contenu. Il utilise des jeux de données de langue modulaires et des tables de n-grammes précalculées pour séparer la logique de détection des données de profil spécifiques à la langue.

Chinese-Vicuna est un grand modèle de langage chinois et une IA suivant des instructions basée sur l'architecture LLaMA. Il est spécifiquement conçu pour la compréhension et la génération de langage naturel en chinois, utilisant un modèle ajusté par instructions pour suivre des invites utilisateur complexes à travers les conversations. Le projet fournit un framework d'ajustement LoRA et des systèmes de quantification pour permettre l'adaptation du modèle et l'inférence sur du matériel grand public. Il implémente l'inférence quantifiée pour réduire l'utilisation de la mémoire sur les CPU et les GPU, soutenu par une implémentation C++ de bas niveau pour minimiser les besoins en ressources système. Le système couvre un large éventail de capacités de traitement du langage naturel, incluant la gestion de conversations multi-tours, la traduction multilingue et la génération de code de programmation. Il inclut également des outils pour l'entraînement spécifique au domaine, la conversion de format de modèle et une interface de chat interactive avec sortie de texte en streaming.

python-goose est une bibliothèque Python pour le web scraping et l'extraction de contenu. Elle fonctionne comme un suppresseur de boilerplate HTML et un analyseur d'articles conçu pour isoler le texte principal et les métadonnées des pages web en supprimant la navigation, le bruit de mise en page et les éléments non essentiels. L'outil propose des capacités de traitement multilingue, utilisant des analyseurs de mots vides (stop-words) spécifiques à la langue pour identifier et extraire le contenu principal à travers différentes langues. Il identifie et collecte également les médias intégrés, y compris les URL sources et les codes d'intégration pour les images et vidéos principales associées à un article. La bibliothèque couvre de vastes domaines fonctionnels, notamment le web scraping automatisé, l'extraction de contenu structuré et la collecte ciblée de médias d'articles.