Awesome GitHub RepositoriesText Processing

ripgrep is a command-line utility designed for searching through large file trees and source code repositories. It functions as a recursive text processor that traverses directories to locate and display matching patterns, serving as a high-performance alternative to traditional search tools. The tool distinguishes itself through a focus on execution speed and intelligent file handling. It utilizes a finite automata-based regular expression engine to ensure linear time complexity and employs hardware-level acceleration for literal byte sequence scanning. By integrating with version control sy

Nanoid is a library for generating unique, fixed-length identifiers designed for distributed systems and database indexing. It produces compact, URL-safe strings by mapping random byte values to a custom character set, allowing for consistent memory allocation and predictable indexing performance across independent nodes without the need for central coordination. The library distinguishes itself by utilizing system-level, cryptographically secure entropy sources to ensure that every generated identifier is statistically unpredictable. This approach provides resistance against collision attack

Colly is a web scraping framework and concurrent crawler written in Go. It provides a system for traversing web pages, following links, and extracting structured data from HTML and XML documents. The framework includes a distributed scraping engine designed to spread data collection tasks across multiple instances to increase throughput. It ensures compliance with website owner policies by automatically reading and respecting robots.txt files. The system manages request lifecycles through domain-based rate limiting, concurrency controls, and session management via a stateful cookie jar. It s

goquery is a Go HTML parsing library and CSS selector engine used to isolate and retrieve specific text or attributes from HTML documents. It functions as an HTML DOM manipulator that converts raw HTML strings into a structured tree for programmatic navigation and search. The library provides a fluent interface for chaining selection and filtering operations and utilizes a wrapper-based abstraction to simplify data extraction and manipulation of nodes. It employs an iterator-based processing mechanism to apply operations to every node within a matched selection. Its primary capabilities cove

Bleve is a search indexing engine library written in Go, designed to provide full-text search and document retrieval capabilities for embedded application data. It functions as a framework for indexing structured or unstructured information, allowing developers to build searchable collections that support complex query logic and data analysis. The engine distinguishes itself through a pluggable analysis pipeline that normalizes text before indexing, alongside support for vector similarity search to identify semantically related content. It utilizes finite-state transducer automata for efficie

ripgrep-all is a command-line utility that extends ripgrep to perform regular expression searches across binary files, compressed archives, and media formats. It functions as a universal text extractor that converts non-plain-text formats, such as PDFs, E-books, and Office documents, into searchable text. The tool uses a system of adapters to transform binary data into plain text and utilizes a local database to cache these extracted versions, accelerating repeated search operations. It identifies file types by analyzing header magic bytes rather than relying on file extensions. The project

YYText is an iOS rich text framework used for rendering and editing complex attributed text. It features an asynchronous text rendering engine that calculates layout on background threads to maintain interface performance, alongside an attributed text serializer for converting rich text structures into storable formats. The framework provides specialized tools for non-rectangular text layout, allowing text to wrap around custom container paths and exclusion shapes. It also supports advanced typography requirements, including vertical text formatting for East Asian languages and the applicatio

This project is a POSIX shell toolset providing utilities for parsing, formatting, interpreting, and validating shell scripts. It includes a parser that converts Bash and Zsh source code into structured syntax trees, a formatter for standardizing script layout, and a runtime interpreter for executing shell commands and script logic. The toolset supports dialect-aware parsing to handle multiple shell language specifications. It provides capabilities for static analysis, allowing shell source code to be converted into structured data for external manipulation or validation against specific dial

i18next is a JavaScript localization library and internationalization framework used to translate application text into multiple languages. It functions as a dynamic translation manager that handles translation resources, language detection, and the mapping of unique keys to localized strings. The system features a pluralization and interpolation engine that applies language-specific grammatical rules for numeric counts and injects dynamic variables into strings. It utilizes a plugin-based backend architecture to load and cache translation namespaces from various sources, including remote ser

Grex is a regular expression generator and Rust pattern library that synthesizes a single regular expression from a set of provided text test cases. It functions as a command-line tool and a library, utilizing a Rust-based engine to analyze commonalities across input strings to create matching patterns. The project distinguishes itself through Unicode-aware grapheme processing, ensuring consistent matching across diverse character sets and non-ASCII text. It also provides Python bindings to make its core Rust logic available within Python environments. The system covers pattern generalizatio

Peco is an interactive text filter and fuzzy finder for the terminal. It serves as a terminal user interface selection tool that filters standard input in real-time using fuzzy matching and regular expressions. The tool preserves and renders ANSI color escape sequences from piped input streams while performing matching logic on plain-text versions. It supports multi-stage filtering, allowing users to freeze result sets to create a new base for subsequent refinements. Capability areas include advanced search filtering with negative matching, multi-item selection, and the ability to pipe selec

DTCoreText is an iOS rich text engine and attributed string converter designed to render formatted HTML content natively. It transforms HTML strings into native text styles and layouts, allowing applications to display rich text without the overhead of a web view. The project functions as a CoreText HTML renderer that integrates HTML strings with the CoreText framework for high-performance rendering. It provides a lightweight alternative for displaying rich text within native application labels and text views. The engine includes an HTML-to-attributed string parser and a native text layout e

FlashText एक डिक्शनरी-आधारित टेक्स्ट प्रोसेसर और कीवर्ड एक्सट्रैक्शन लाइब्रेरी है जिसे उच्च-प्रदर्शन स्ट्रिंग सर्चिंग और प्रोग्रामेटिक सामग्री प्रतिस्थापन के लिए डिज़ाइन किया गया है। यह टेक्स्ट से पूर्व-निर्धारित शब्दों की पहचान करने और उन्हें पुनः प्राप्त करने या डेटा की बड़ी मात्रा में विशिष्ट शब्दों को प्रतिस्थापन स्ट्रिंग्स के साथ बदलने के लिए विशेष यूटिलिटीज प्रदान करती है। यह प्रोजेक्ट मल्टी-पैटर्न मैचिंग को सक्षम करने के लिए Aho-Corasick स्ट्रिंग सर्च कार्यान्वयन का उपयोग करता है। यह दृष्टिकोण सिंगल-पास टेक्स्ट स्कैनिंग और लीनियर-टाइम कॉम्प्लेक्सिटी सर्च की अनुमति देता है, यह सुनिश्चित करता है कि प्रोसेसिंग समय कीवर्ड डिक्शनरी के साइज़ के बजाय इनपुट टेक्स्ट की लंबाई पर निर्भर करता है। यह लाइब्रेरी लक्षित शब्द सूचियों को बनाए रखने के लिए कीवर्ड डिक्शनरी मैनेजमेंट को कवर करती है और ऑटोमेटेड टेक्स्ट रिप्लेसमेंट और कीवर्ड एक्सट्रैक्शन दोनों के लिए वर्कफ़्लो प्रदान करती है। ये क्षमताएं विशाल डेटासेट्स के भीतर पूर्व-निर्धारित शब्दों के व्यवस्थित संशोधन और पुनर्प्राप्ति की अनुमति देती हैं।

Blackfriday is a Go library for parsing and converting Markdown text into HTML, LaTeX, and other structured formats. It functions as an extensible Markdown processor that transforms syntax into target markup languages. The project is distinguished by its pluggable rendering architecture, which allows for the production of diverse output targets such as Slack message styles, Confluence Wiki Markup, and GitHub Flavored Markdown. It supports custom syntax extensions including definition lists, footnotes, autolinks, and strikethroughs. The processor includes utilities for generating automatic ta

PhoneNumberKit is a Swift framework for parsing, formatting, and validating international phone numbers according to global numbering standards. It provides a system of lookup tables, structural parsers, and validation tools to handle the complexities of international telephone numbering. The library converts raw strings into structured objects containing country codes and national numbers. It translates international dialing codes into full country names and verifies whether phone number strings are logically possible and conform to specific regional numbering plans. The project includes ut

python-pinyin is a Python library for transliterating simplified and traditional Chinese characters into phonetic pinyin. It functions as a transliteration system that converts text while supporting tone sandhi and providing utilities to transform pinyin between different formats, such as numeric tones, accent marks, or phonetic initials. The library features a polyphonic character resolver that analyzes surrounding word context to select the correct pronunciation for characters with multiple sounds. It also includes a customizable dictionary system that allows the extension of default transl

Chroma is a syntax highlighter written in Go that converts raw source code into colorized text or HTML. It identifies programming languages and tokens using language-specific lexers to produce highlighted output for various environments. The project utilizes a theme-driven colorization system that decouples token types from final colors through an intermediate style mapping layer. This allows for custom code theming by mapping specific token categories to foreground and background colors. The system covers language identification, terminal text formatting for ANSI-compatible viewers, and web

Goldmark, Go में लिखा गया एक Markdown पार्सर और रेंडरर है जो Markdown टेक्स्ट को HTML या XHTML में बदलता है। यह सुसंगत पार्सिंग व्यवहार सुनिश्चित करने के लिए CommonMark विनिर्देश को लागू करता है। यह प्रोजेक्ट एक मॉड्यूलर एक्सटेंशन रजिस्ट्री का उपयोग करता है, जो टेबल, टास्क लिस्ट और फुटनोट्स जैसे कस्टम सिंटैक्स को जोड़ने की अनुमति देता है। इसमें चीनी, जापानी और कोरियाई वर्णों के लिए विशिष्ट लाइन-ब्रेकिंग और जोर (emphasis) नियमों को संभालने के लिए CJK भाषाओं के लिए विशेष समर्थन भी शामिल है। यह लाइब्रेरी प्रोग्रामेटिक डॉक्यूमेंट विश्लेषण और संशोधन के लिए टेक्स्ट को एक एब्स्ट्रैक्ट सिंटैक्स ट्री में पार्स करती है। इसकी क्षमताओं में URLs और ईमेल की ऑटोमैटिक लिंकिंग, टाइपोग्राफिक पंक्चुएशन फॉर्मेटिंग और एलिमेंट्स को कस्टम एट्रिब्यूट्स असाइन करना शामिल है।

This is a TOML parser and serializer for the Go language. It serves as a data serialization library and configuration file mapper that encodes and decodes data between Go structures and the TOML configuration format. The library provides interfaces for custom type marshaling, allowing for specialized logic when parsing or serializing specific data types. It transforms structured objects into deterministic TOML documents for storage or transmission. The project covers a broad range of data processing capabilities, including structured value encoding, TOML data generation, and metadata inspect

go-humanize एक Go लाइब्रेरी है जो रॉ नंबर्स, बाइट साइज़ और ड्यूरेशन को नेचुरल लैंग्वेज स्ट्रिंग्स में बदलने के लिए है ताकि उन्हें पढ़ने में आसानी हो। यह प्रोजेक्ट SI या IEC स्टैंडर्ड्स का उपयोग करके बाइट साइज़ के लिए विशेष फॉर्मेटर्स और एक रिलेटिव टाइम फॉर्मेटर प्रदान करता है जो ड्यूरेशन को 'तीन दिन पहले' जैसे विवरणों में बदल देता है। इसमें संख्यात्मक मानों को इंटरनेशनल सिस्टम ऑफ यूनिट्स के प्रीफिक्स और सिंबल में बदलने के लिए एक SI यूनिट फॉर्मेटर भी शामिल है। तकनीकी इकाइयों से परे, लाइब्रेरी में संज्ञाओं को बहुवचन बनाने, ऑर्डिनल नंबर्स उत्पन्न करने और व्याकरणिक कंजंक्शन के साथ शब्द सूचियों को जोड़ने के लिए अंग्रेजी नेचुरल लैंग्वेज यूटिलिटीज शामिल हैं। अतिरिक्त संख्यात्मक उपकरण हज़ारों के सेपरेटर, दशमलव परिशुद्धता और बड़ी संख्या के सफिक्स को संभालते हैं।