Awesome GitHub RepositoriesText Segmentation

FFmpeg is a cross-platform multimedia framework designed for the recording, conversion, and streaming of audio and video content. It functions as a comprehensive toolkit that provides both a command-line utility for direct media manipulation and a collection of low-level libraries for integration into custom applications. At its core, the project utilizes a packet-based stream engine and a format-agnostic abstraction layer to handle diverse media standards, containers, and network protocols. The framework distinguishes itself through a modular, graph-based filter execution model that allows f

Pretext is a canvas-based text layout engine designed to calculate precise text dimensions and line breaks for custom rendering. It serves as a rich text measurement tool and a cross-browser typography normalizer, enabling the determination of pixel-perfect widths and heights for mixed inline content without relying on browser CSS. The project distinguishes itself through its ability to handle complex typography and dynamic layouts. It implements language-specific segmentation rules for CJK and Hangul scripts and corrects emoji width variances between DOM and canvas rendering. Additionally, i

This project is a Chinese text segmentation library and tokenizer designed to split Chinese sentences into individual words. It serves as a natural language processing tool for splitting characters into words, tagging parts of speech, and extracting keywords using statistical analysis. The library distinguishes itself through support for custom dictionary configuration and vocabulary file management, allowing users to override default segmentation rules for domain-specific accuracy. It also includes a TF-IDF keyword extractor to identify significant words and core topics within documents. Th

spaCy is a Python natural language processing framework designed for industrial-scale text processing. It converts raw text into structured data for machine learning pipelines through a combination of statistical language model trainers, transformer-based text processors, and syntactic dependency parsers. The project enables the integration of pretrained transformer architectures to perform complex linguistic analysis and multi-task learning. It also provides a specialized system for neural named entity recognition to identify and categorize key entities within text. The framework covers a b

This project is a Python framework for building autonomous, event-driven agent systems. It provides a unified runtime for orchestrating multi-agent workflows, managing persistent conversation state, and executing code within secure, isolated sandbox environments. The framework is designed to handle complex task delegation, allowing agents to invoke other agents as tools while maintaining context across multi-turn interactions. The framework distinguishes itself through its deep integration with the Model Context Protocol, enabling agents to connect to external data sources and remote services

xi-editor is a high-performance text editor core written in Rust. It employs a client-server architecture that separates the backend editor logic from the user interface, allowing diverse frontends to communicate with the core via a standardized protocol. The project is distinguished by its use of rope-based text buffers for efficient manipulation of large documents and a collaborative engine powered by conflict-free replicated data types to synchronize concurrent edits. It further features an extensible plugin system that integrates external binaries and third-party tools through JSON-based

VideoLingo is an automated video localization suite designed to transcribe, translate, and dub video content. It functions as a translation pipeline that utilizes large language models to convert spoken audio into precise text segments and translate them into multiple languages. The system differentiates itself through a multi-step translation refinement process and a specialized natural language processing utility that segments text into single-line captions meeting broadcast standards. It also integrates synthetic voiceover generation to replace or augment original audio tracks. The projec

Analysis-ik is a Chinese text segmenter and analysis plugin for Lucene-based search engines. It provides a specialized analyzer for splitting Chinese sentences into meaningful words to improve indexing and search accuracy within Elasticsearch and OpenSearch. The project features a dynamic dictionary manager that can load word libraries and stop-word files from remote HTTP endpoints. It monitors metadata headers on these remote files to trigger automatic vocabulary updates without requiring a service restart. The analyzer supports both fine-grained exhaustive and coarse-grained smart segmenta

llmware is a Python framework for AI agent orchestration and model management, designed to coordinate multi-model workflows and autonomous agents. It provides a unified model catalog and standardized interface to execute specialized language models for complex research, analysis, and structured data generation. The project distinguishes itself through its heavy emphasis on local execution and quantized inference, allowing models to run on private infrastructure using CPU, GPU, and NPU acceleration via runtimes like ONNX and OpenVino. It features a specialized ability to translate natural lang

Easy-dataset is a comprehensive platform designed for the end-to-end management of machine learning datasets, specifically tailored for language and vision model fine-tuning. It functions as a centralized environment for the entire data lifecycle, encompassing the automated generation of synthetic training data, the structural organization of document collections, and the systematic annotation of individual data points. The platform distinguishes itself through its integrated evaluation and orchestration capabilities. It provides a dedicated suite for benchmarking models, featuring blind side

This project is a multimodal translation framework and large language model capable of speech-to-speech, speech-to-text, and text-to-text translation across nearly 100 languages. It provides a real-time speech translation engine and a comprehensive toolkit for converting spoken audio between languages. The system is distinguished by its ability to preserve the original speaker's tone, pace, and prosody during translation. It utilizes a specialized on-device inference toolkit that converts model checkpoints into C-based libraries, enabling low-latency execution on mobile and edge hardware with

SentencePiece is a text segmentation engine and tokenization library designed for machine learning workflows. It provides a comprehensive toolkit for transforming raw text into subword units or numerical identifiers, enabling consistent data representation for neural network training and inference. The library supports the training of segmentation models from raw text, allowing for the creation of custom vocabularies tailored to specific domain requirements. The project distinguishes itself through its byte-level encoding and fallback mechanisms, which ensure that every input can be represent

Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.

This project is a collection of supervised and unsupervised machine learning algorithms implemented from scratch using Python. It serves as an educational resource for studying model training, parameter optimization, and the implementation of core predictive models. The library provides a variety of supervised learning tools, including linear and logistic regression, decision trees, and support vector machines. It also features unsupervised learning capabilities for discovering patterns in unlabeled datasets through clustering algorithms. Broad capability areas include ensemble learning thro

OpenCC is a library and command-line tool for converting text between Simplified Chinese, Traditional Chinese, and Japanese Kanji. It operates at both the individual character and multi-character phrase levels, and applies region-specific vocabulary choices for Mainland China, Taiwan, and Hong Kong during conversion. The conversion engine resolves ambiguous character mappings using semantic and contextual rules, normalizes variant character forms for consistent orthography, and sequences multiple dictionary files into a configurable pipeline. It supports embedding custom conversion rules dire

ytDownloader is a video downloader and media extraction tool that uses the yt-dlp engine to retrieve video and audio files from various social media and video sharing platforms. It functions as a utility for capturing full media files, specific segments or ranges of tracks, and entire video playlists. The project includes a hardware-accelerated video compressor to reduce file sizes while maintaining visual quality. It also features a subtitle downloader capable of retrieving both text captions and embedded subtitle tracks for accessibility and translation. The system handles broad media task

This repository is a comprehensive collection of reference implementations and sample libraries for the Universal Windows Platform. It provides practical examples of how to use Windows Runtime APIs to build cross-device applications, including detailed guidance on XAML-based declarative user interfaces and DirectX-integrated rendering. The project distinguishes itself by providing a wide array of hardware integration suites, covering low-level communication with USB, Serial, I2C, SPI, and GPIO peripherals. It includes specialized implementations for mixed reality holographic rendering, advanc

imewlconverter is an input method editor wordlist converter and format transformer designed to migrate user dictionaries and phrase lists between different software environments. It functions as a cross-platform dictionary migrator, translating proprietary binary and text wordlists for use across Windows, macOS, and mobile systems. The tool standardizes diverse lexicon formats, such as WL, FIT, DCTX, LD2, and QPYD, into common structures to ensure cross-platform compatibility. It specifically handles binary wordlist extraction and the transformation of custom phrase lists for systems includin

Any-rule is a multi-platform regular expression tool that provides a curated catalog of over 70 ready-to-use patterns for validating and extracting common data formats. The project separates its static regex collection from editor-specific plugins, allowing the same pattern library to be accessed through VS Code, IntelliJ IDEA, Alfred Workflow, and a web interface. The tool enables keyword-based pattern retrieval, letting users search for the correct regex by typing descriptive terms rather than remembering exact syntax. It covers a broad range of validation needs including email addresses, U

OfficeCLI ist eine Headless-Office-Suite und ein Automatisierungswerkzeug, das darauf ausgelegt ist, Microsoft Office-Dokumente programmatisch zu lesen, zu bearbeiten und zu generieren. Es fungiert als OOXML-Manipulationsbibliothek und Dokument-Templating-Engine und bietet eine eigenständige Binärdatei, die die Verwaltung von Word-, Excel- und PowerPoint-Dateien ermöglicht, ohne dass eine lokale Installation von Office-Software erforderlich ist. Das Projekt zeichnet sich dadurch aus, dass es Dokumentoperationen als Werkzeuge für KI-Agenten über einen JSON-RPC-Server und das Model Context Protocol offenlegt. Es ermöglicht fortgeschrittene Anpassungen durch rohe XML-Manipulation mittels XPath und bietet ein Serialisierungssystem, das Dokument-Subtrees in wiederholbare JSON-Batches ausgibt. Das Tool deckt ein breites Spektrum an Funktionen ab, einschließlich programmatischem Tabellenkalkulations-Engineering mit Formelauswertung und Pivot-Tabellen-Generierung sowie umfassenden Textverarbeitungsaufgaben wie Stilverwaltung, Revisionsverfolgung und mehrsprachiger Textformatierung. Es enthält zudem Utilities für Datenvisualisierung, Inhaltsextraktion in strukturiertes JSON oder High-Fidelity-HTML sowie das Zusammenführen von JSON-Daten in vordefinierte Templates für die automatisierte Berichterstellung.