Awesome GitHub RepositoriesText Segmentation

FFmpeg is a cross-platform multimedia framework designed for the recording, conversion, and streaming of audio and video content. It functions as a comprehensive toolkit that provides both a command-line utility for direct media manipulation and a collection of low-level libraries for integration into custom applications. At its core, the project utilizes a packet-based stream engine and a format-agnostic abstraction layer to handle diverse media standards, containers, and network protocols. The framework distinguishes itself through a modular, graph-based filter execution model that allows f

Pretext is a canvas-based text layout engine designed to calculate precise text dimensions and line breaks for custom rendering. It serves as a rich text measurement tool and a cross-browser typography normalizer, enabling the determination of pixel-perfect widths and heights for mixed inline content without relying on browser CSS. The project distinguishes itself through its ability to handle complex typography and dynamic layouts. It implements language-specific segmentation rules for CJK and Hangul scripts and corrects emoji width variances between DOM and canvas rendering. Additionally, i

This project is a Chinese text segmentation library and tokenizer designed to split Chinese sentences into individual words. It serves as a natural language processing tool for splitting characters into words, tagging parts of speech, and extracting keywords using statistical analysis. The library distinguishes itself through support for custom dictionary configuration and vocabulary file management, allowing users to override default segmentation rules for domain-specific accuracy. It also includes a TF-IDF keyword extractor to identify significant words and core topics within documents. Th

spaCy is a Python natural language processing framework designed for industrial-scale text processing. It converts raw text into structured data for machine learning pipelines through a combination of statistical language model trainers, transformer-based text processors, and syntactic dependency parsers. The project enables the integration of pretrained transformer architectures to perform complex linguistic analysis and multi-task learning. It also provides a specialized system for neural named entity recognition to identify and categorize key entities within text. The framework covers a b

This project is a Python framework for building autonomous, event-driven agent systems. It provides a unified runtime for orchestrating multi-agent workflows, managing persistent conversation state, and executing code within secure, isolated sandbox environments. The framework is designed to handle complex task delegation, allowing agents to invoke other agents as tools while maintaining context across multi-turn interactions. The framework distinguishes itself through its deep integration with the Model Context Protocol, enabling agents to connect to external data sources and remote services

xi-editor is a high-performance text editor core written in Rust. It employs a client-server architecture that separates the backend editor logic from the user interface, allowing diverse frontends to communicate with the core via a standardized protocol. The project is distinguished by its use of rope-based text buffers for efficient manipulation of large documents and a collaborative engine powered by conflict-free replicated data types to synchronize concurrent edits. It further features an extensible plugin system that integrates external binaries and third-party tools through JSON-based

VideoLingo is an automated video localization suite designed to transcribe, translate, and dub video content. It functions as a translation pipeline that utilizes large language models to convert spoken audio into precise text segments and translate them into multiple languages. The system differentiates itself through a multi-step translation refinement process and a specialized natural language processing utility that segments text into single-line captions meeting broadcast standards. It also integrates synthetic voiceover generation to replace or augment original audio tracks. The projec

Analysis-ik is a Chinese text segmenter and analysis plugin for Lucene-based search engines. It provides a specialized analyzer for splitting Chinese sentences into meaningful words to improve indexing and search accuracy within Elasticsearch and OpenSearch. The project features a dynamic dictionary manager that can load word libraries and stop-word files from remote HTTP endpoints. It monitors metadata headers on these remote files to trigger automatic vocabulary updates without requiring a service restart. The analyzer supports both fine-grained exhaustive and coarse-grained smart segmenta

llmware is a Python framework for AI agent orchestration and model management, designed to coordinate multi-model workflows and autonomous agents. It provides a unified model catalog and standardized interface to execute specialized language models for complex research, analysis, and structured data generation. The project distinguishes itself through its heavy emphasis on local execution and quantized inference, allowing models to run on private infrastructure using CPU, GPU, and NPU acceleration via runtimes like ONNX and OpenVino. It features a specialized ability to translate natural lang

Easy-dataset is a comprehensive platform designed for the end-to-end management of machine learning datasets, specifically tailored for language and vision model fine-tuning. It functions as a centralized environment for the entire data lifecycle, encompassing the automated generation of synthetic training data, the structural organization of document collections, and the systematic annotation of individual data points. The platform distinguishes itself through its integrated evaluation and orchestration capabilities. It provides a dedicated suite for benchmarking models, featuring blind side

This project is a multimodal translation framework and large language model capable of speech-to-speech, speech-to-text, and text-to-text translation across nearly 100 languages. It provides a real-time speech translation engine and a comprehensive toolkit for converting spoken audio between languages. The system is distinguished by its ability to preserve the original speaker's tone, pace, and prosody during translation. It utilizes a specialized on-device inference toolkit that converts model checkpoints into C-based libraries, enabling low-latency execution on mobile and edge hardware with

SentencePiece is a text segmentation engine and tokenization library designed for machine learning workflows. It provides a comprehensive toolkit for transforming raw text into subword units or numerical identifiers, enabling consistent data representation for neural network training and inference. The library supports the training of segmentation models from raw text, allowing for the creation of custom vocabularies tailored to specific domain requirements. The project distinguishes itself through its byte-level encoding and fallback mechanisms, which ensure that every input can be represent

Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.

This project is a collection of supervised and unsupervised machine learning algorithms implemented from scratch using Python. It serves as an educational resource for studying model training, parameter optimization, and the implementation of core predictive models. The library provides a variety of supervised learning tools, including linear and logistic regression, decision trees, and support vector machines. It also features unsupervised learning capabilities for discovering patterns in unlabeled datasets through clustering algorithms. Broad capability areas include ensemble learning thro

OpenCC is a library and command-line tool for converting text between Simplified Chinese, Traditional Chinese, and Japanese Kanji. It operates at both the individual character and multi-character phrase levels, and applies region-specific vocabulary choices for Mainland China, Taiwan, and Hong Kong during conversion. The conversion engine resolves ambiguous character mappings using semantic and contextual rules, normalizes variant character forms for consistent orthography, and sequences multiple dictionary files into a configurable pipeline. It supports embedding custom conversion rules dire

ytDownloader is a video downloader and media extraction tool that uses the yt-dlp engine to retrieve video and audio files from various social media and video sharing platforms. It functions as a utility for capturing full media files, specific segments or ranges of tracks, and entire video playlists. The project includes a hardware-accelerated video compressor to reduce file sizes while maintaining visual quality. It also features a subtitle downloader capable of retrieving both text captions and embedded subtitle tracks for accessibility and translation. The system handles broad media task

This repository is a comprehensive collection of reference implementations and sample libraries for the Universal Windows Platform. It provides practical examples of how to use Windows Runtime APIs to build cross-device applications, including detailed guidance on XAML-based declarative user interfaces and DirectX-integrated rendering. The project distinguishes itself by providing a wide array of hardware integration suites, covering low-level communication with USB, Serial, I2C, SPI, and GPIO peripherals. It includes specialized implementations for mixed reality holographic rendering, advanc

imewlconverter is an input method editor wordlist converter and format transformer designed to migrate user dictionaries and phrase lists between different software environments. It functions as a cross-platform dictionary migrator, translating proprietary binary and text wordlists for use across Windows, macOS, and mobile systems. The tool standardizes diverse lexicon formats, such as WL, FIT, DCTX, LD2, and QPYD, into common structures to ensure cross-platform compatibility. It specifically handles binary wordlist extraction and the transformation of custom phrase lists for systems includin

Any-rule is a multi-platform regular expression tool that provides a curated catalog of over 70 ready-to-use patterns for validating and extracting common data formats. The project separates its static regex collection from editor-specific plugins, allowing the same pattern library to be accessed through VS Code, IntelliJ IDEA, Alfred Workflow, and a web interface. The tool enables keyword-based pattern retrieval, letting users search for the correct regex by typing descriptive terms rather than remembering exact syntax. It covers a broad range of validation needs including email addresses, U

OfficeCLI est une suite bureautique headless et un outil d'automatisation conçu pour lire, éditer et générer par programmation des documents Microsoft Office. Il fonctionne comme une bibliothèque de manipulation OOXML et un moteur de modélisation de documents, fournissant un binaire autonome qui permet la gestion de fichiers Word, Excel et PowerPoint sans nécessiter une installation locale de logiciels de bureautique. Le projet se distingue en exposant les opérations de document comme des outils pour les agents IA via un serveur JSON-RPC et le protocole Model Context Protocol. Il permet une personnalisation avancée via la manipulation XML brute en utilisant XPath et fournit un système de sérialisation qui déverse les sous-arbres de documents dans des lots JSON rejouables. L'outil couvre un large éventail de capacités, y compris l'ingénierie de feuilles de calcul programmatique avec évaluation de formules et génération de tableaux croisés dynamiques, ainsi que des tâches de traitement de texte complètes telles que la gestion des styles, le suivi des révisions et le formatage de texte multilingue. Il inclut également des utilitaires pour la visualisation de données, l'extraction de contenu en JSON structuré ou HTML haute fidélité, et la fusion de données JSON dans des modèles prédéfinis pour la génération automatisée de rapports.