Awesome GitHub RepositoriesText Processing

ripgrep is a command-line utility designed for searching through large file trees and source code repositories. It functions as a recursive text processor that traverses directories to locate and display matching patterns, serving as a high-performance alternative to traditional search tools. The tool distinguishes itself through a focus on execution speed and intelligent file handling. It utilizes a finite automata-based regular expression engine to ensure linear time complexity and employs hardware-level acceleration for literal byte sequence scanning. By integrating with version control sy

Nanoid is a library for generating unique, fixed-length identifiers designed for distributed systems and database indexing. It produces compact, URL-safe strings by mapping random byte values to a custom character set, allowing for consistent memory allocation and predictable indexing performance across independent nodes without the need for central coordination. The library distinguishes itself by utilizing system-level, cryptographically secure entropy sources to ensure that every generated identifier is statistically unpredictable. This approach provides resistance against collision attack

Colly is a web scraping framework and concurrent crawler written in Go. It provides a system for traversing web pages, following links, and extracting structured data from HTML and XML documents. The framework includes a distributed scraping engine designed to spread data collection tasks across multiple instances to increase throughput. It ensures compliance with website owner policies by automatically reading and respecting robots.txt files. The system manages request lifecycles through domain-based rate limiting, concurrency controls, and session management via a stateful cookie jar. It s

goquery is a Go HTML parsing library and CSS selector engine used to isolate and retrieve specific text or attributes from HTML documents. It functions as an HTML DOM manipulator that converts raw HTML strings into a structured tree for programmatic navigation and search. The library provides a fluent interface for chaining selection and filtering operations and utilizes a wrapper-based abstraction to simplify data extraction and manipulation of nodes. It employs an iterator-based processing mechanism to apply operations to every node within a matched selection. Its primary capabilities cove

Bleve is a search indexing engine library written in Go, designed to provide full-text search and document retrieval capabilities for embedded application data. It functions as a framework for indexing structured or unstructured information, allowing developers to build searchable collections that support complex query logic and data analysis. The engine distinguishes itself through a pluggable analysis pipeline that normalizes text before indexing, alongside support for vector similarity search to identify semantically related content. It utilizes finite-state transducer automata for efficie

ripgrep-all is a command-line utility that extends ripgrep to perform regular expression searches across binary files, compressed archives, and media formats. It functions as a universal text extractor that converts non-plain-text formats, such as PDFs, E-books, and Office documents, into searchable text. The tool uses a system of adapters to transform binary data into plain text and utilizes a local database to cache these extracted versions, accelerating repeated search operations. It identifies file types by analyzing header magic bytes rather than relying on file extensions. The project

YYText is an iOS rich text framework used for rendering and editing complex attributed text. It features an asynchronous text rendering engine that calculates layout on background threads to maintain interface performance, alongside an attributed text serializer for converting rich text structures into storable formats. The framework provides specialized tools for non-rectangular text layout, allowing text to wrap around custom container paths and exclusion shapes. It also supports advanced typography requirements, including vertical text formatting for East Asian languages and the applicatio

This project is a POSIX shell toolset providing utilities for parsing, formatting, interpreting, and validating shell scripts. It includes a parser that converts Bash and Zsh source code into structured syntax trees, a formatter for standardizing script layout, and a runtime interpreter for executing shell commands and script logic. The toolset supports dialect-aware parsing to handle multiple shell language specifications. It provides capabilities for static analysis, allowing shell source code to be converted into structured data for external manipulation or validation against specific dial

i18next is a JavaScript localization library and internationalization framework used to translate application text into multiple languages. It functions as a dynamic translation manager that handles translation resources, language detection, and the mapping of unique keys to localized strings. The system features a pluralization and interpolation engine that applies language-specific grammatical rules for numeric counts and injects dynamic variables into strings. It utilizes a plugin-based backend architecture to load and cache translation namespaces from various sources, including remote ser

Grex is a regular expression generator and Rust pattern library that synthesizes a single regular expression from a set of provided text test cases. It functions as a command-line tool and a library, utilizing a Rust-based engine to analyze commonalities across input strings to create matching patterns. The project distinguishes itself through Unicode-aware grapheme processing, ensuring consistent matching across diverse character sets and non-ASCII text. It also provides Python bindings to make its core Rust logic available within Python environments. The system covers pattern generalizatio

Peco is an interactive text filter and fuzzy finder for the terminal. It serves as a terminal user interface selection tool that filters standard input in real-time using fuzzy matching and regular expressions. The tool preserves and renders ANSI color escape sequences from piped input streams while performing matching logic on plain-text versions. It supports multi-stage filtering, allowing users to freeze result sets to create a new base for subsequent refinements. Capability areas include advanced search filtering with negative matching, multi-item selection, and the ability to pipe selec

DTCoreText is an iOS rich text engine and attributed string converter designed to render formatted HTML content natively. It transforms HTML strings into native text styles and layouts, allowing applications to display rich text without the overhead of a web view. The project functions as a CoreText HTML renderer that integrates HTML strings with the CoreText framework for high-performance rendering. It provides a lightweight alternative for displaying rich text within native application labels and text views. The engine includes an HTML-to-attributed string parser and a native text layout e

FlashText 是一个基于字典的文本处理器和关键词提取库，专为高性能字符串搜索和程序化内容替换而设计。它提供专门的实用程序，用于从文本中识别和检索预定义术语，或在海量数据中将特定术语替换为其他字符串。 该项目利用 Aho-Corasick 字符串搜索实现来支持多模式匹配。这种方法允许进行单遍文本扫描和线性时间复杂度搜索，确保处理时间取决于输入文本的长度，而不是关键词字典的大小。 该库涵盖了用于维护目标术语列表的关键词字典管理，并提供了自动化文本替换和关键词提取的工作流。这些功能允许在海量数据集中系统地修改和检索预定义术语。

Blackfriday is a Go library for parsing and converting Markdown text into HTML, LaTeX, and other structured formats. It functions as an extensible Markdown processor that transforms syntax into target markup languages. The project is distinguished by its pluggable rendering architecture, which allows for the production of diverse output targets such as Slack message styles, Confluence Wiki Markup, and GitHub Flavored Markdown. It supports custom syntax extensions including definition lists, footnotes, autolinks, and strikethroughs. The processor includes utilities for generating automatic ta

PhoneNumberKit 是一个 Swift 框架，用于根据全球编号标准解析、格式化和验证国际电话号码。它提供了一套查找表、结构化解析器和验证工具，以处理国际电话编号的复杂性。 该库将原始字符串转换为包含国家代码和国内号码的结构化对象。它将国际拨号代码转换为完整的国家名称，并验证电话号码字符串在逻辑上是否可能，以及是否符合特定的区域编号计划。 该项目包含用于将原始数字转换为标准化国际或国内布局的实用程序。它还提供了实时用户输入掩码功能，以及在用户输入时显示区域特定占位符和国旗的功能。

python-pinyin 是一个 Python 库，用于将简体和繁体中文转换为拼音。它作为一个音译系统，在转换文本的同时支持多音字处理，并提供将拼音转换为不同格式（如数字声调、声调符号或声母）的实用程序。 该库具有多音字解析器，可分析周围词语上下文以选择正确的发音。它还包括一个可自定义的字典系统，允许使用自定义的单字和词组拼音映射来扩展默认的音译结果。 该工具集涵盖了更广泛的语言操作，包括提取声母和韵母等语音成分、应用各种拼音风格以及处理不可转换字符。这些功能既可作为库使用，也可通过命令行界面进行文本转换和声调格式转换。

Chroma 是一个用 Go 编写的语法高亮器，可将原始源代码转换为彩色文本或 HTML。它使用特定于语言的词法分析器来识别编程语言和标记，从而为各种环境生成高亮输出。 该项目利用基于主题的着色系统，通过中间样式映射层将标记类型与最终颜色解耦。这允许通过将特定的标记类别映射到前景色和背景色来进行自定义代码主题设置。 该系统涵盖了语言识别、用于 ANSI 兼容查看器的终端文本格式化，以及通过带有行号和 CSS 类的 HTML 生成进行的 Web 代码呈现。它还包括一个用于直接在终端中为文件着色的命令行界面。

Goldmark 是一个用 Go 编写的 Markdown 解析器和渲染器，将 Markdown 文本转换为 HTML 或 XHTML。它实现了 CommonMark 规范，以确保解析行为的一致性。 该项目利用模块化扩展注册表，允许添加自定义语法，如表格、任务列表和脚注。它还包括对 CJK（中日韩）语言的专门支持，以处理特定的换行和强调规则。 该库将文本解析为抽象语法树，以便进行程序化的文档分析和修改。其功能包括 URL 和电子邮件的自动链接、排版标点格式化以及为元素分配自定义属性。

This is a TOML parser and serializer for the Go language. It serves as a data serialization library and configuration file mapper that encodes and decodes data between Go structures and the TOML configuration format. The library provides interfaces for custom type marshaling, allowing for specialized logic when parsing or serializing specific data types. It transforms structured objects into deterministic TOML documents for storage or transmission. The project covers a broad range of data processing capabilities, including structured value encoding, TOML data generation, and metadata inspect

go-humanize is a Go library for converting raw numbers, byte sizes, and durations into natural language strings to improve human readability. The project provides specialized formatters for byte sizes using SI or IEC standards and a relative time formatter that transforms durations into descriptions such as three days ago. It also includes an SI unit formatter for converting numeric values to and from International System of Units prefixes and symbols. Beyond technical units, the library includes English natural language utilities for pluralizing nouns, generating ordinal numbers, and joinin