Awesome GitHub RepositoriesString Tokenization

Cleave.js is a JavaScript library that applies real-time visual formatting to form fields as users type. It provides specialized utilities to enforce visual patterns for credit cards, phone numbers, dates, times, and large numerals. The library includes predefined formatting rules for common data types, such as detecting credit card issuers to apply correct spacing and managing country-specific regional patterns for international phone numbers. It also supports large numeral grouping using Western, Indian, and Chinese numbering systems. Users can define custom input patterns using specific b

Compromise is a natural language processing library and rule-based engine designed for English text manipulation, analysis, and parsing. It provides a toolkit for tokenizing text, identifying parts of speech, and performing linguistic analysis to achieve semantic understanding of unstructured strings. The project distinguishes itself through its ability to programmatically transform grammar, such as modifying verb tenses, noun plurality, and adjective forms. It also functions as a named entity recognizer capable of extracting people, places, organizations, dates, and contact information from

This project is a regular expression lexer library and lexical analysis engine used to break input strings into typed token streams. It serves as a foundational component for constructing compilers or interpreters by identifying and categorizing substrings into discrete tokens. The library provides a token stream navigator featuring a cursor-based interface. This allows for sequential traversal of tokenized input and non-destructive lookahead, enabling the inspection of future tokens without advancing the internal position pointer. It includes specific support for recursive descent parsing t

Natural is a natural language processing library for Node.js that provides tools for text analysis, tokenization, and phonetic matching. It functions as a collection of specialized toolsets for word stemming, string similarity quantification, and pattern-based text classification. The library includes a phonetic sound analyzer that converts words into phonetic representations to identify matches based on sound rather than literal spelling. It also features a text classification engine that assigns categories to text inputs using trained models and pattern recognition. Additional capabilities

Fuzzywuzzy is a Python library and text processing utility designed to calculate similarity scores between strings. It functions as a text similarity scoring engine and an approximate string matching tool used to identify the closest textual matches within a list of candidate strings. The library provides a suite of tools for measuring the degree of similarity between pieces of text, accounting for typos and formatting differences. These capabilities include extracting the best match from a candidate list and performing fuzzy string matching through various scoring methods. The toolset cover

This repository contains the HTML specification, which defines the core standards for web page structuring, content organization, and document rendering. It establishes the fundamental algorithms for state-machine-based tokenization, tree construction for the document object model, and origin-based security isolation. The specification provides a framework for defining custom elements with independent lifecycles and registries. It also details the requirements for cross-document communication, session history management, and the synchronization of interface properties with content attributes.

path-to-regexp 是一个 HTTP 路由匹配器和正则表达式生成器，将参数化路径字符串转换为正则表达式，用于匹配 Web 地址。它作为一个 URL 路由解析器和模式编译器，能够将人类可读的路径模板转换为用于路由的功能表达式。 该项目提供了根据定义的路径模式从 URL 中提取命名参数和通配符的功能。它还处理动态路径生成的反向过程，通过将值注入定义的参数占位符，将键值对转换为格式化的路径字符串。 该库涵盖了更广泛的路由实用程序，包括基于令牌的路径解析以及定义自定义参数分隔符的能力。它还支持锚点控制的匹配，以确定正则表达式是否必须从输入字符串的开头或结尾开始匹配。

This project is a PHP implementation of a CSS selector engine that transforms CSS selector strings into compatible XPath expressions for locating elements within documents. It serves as a converter and expression generator that maps CSS selection logic to the XPath query language. The library processes selectors through a pipeline involving lexer-based tokenization and recursive descent parsing to create an abstract syntax tree. It utilizes pattern-matching logic to handle child and sibling relationships, translating CSS pseudo-classes and selectors into functional XPath logic. These capabil

JSON5 is a parser and serializer for a human-readable configuration format that extends JSON. It serves as a JavaScript-based data parser that allows for a more flexible version of the JSON specification to simplify manual editing of data files. The project provides capabilities to support comments, trailing commas, and multi-line strings. It includes utilities to convert this extended syntax into standard JSON for compatibility with tools requiring strict specifications. The library covers data serialization, string parsing, and structural syntax validation. It also provides integration for

js-yaml is a JavaScript library providing a programmatic interface for parsing and dumping YAML data. It functions as a parser and serializer that converts YAML strings into JavaScript objects and transforms JavaScript objects back into YAML format. The library includes a command-line interface for parsing or dumping YAML data via direct input or data pipes. It also features an abstract syntax tree transformer to modify the structure of data during serialization. The project provides capabilities for multi-document processing and the definition of custom schemas and tags to handle specialize

Rainmeter is a Windows desktop widget engine that renders customizable skins and interactive widgets directly on the desktop, supporting live data feeds and user interaction. It functions as a desktop customization platform and skin authoring framework, allowing users to create widgets by defining data sources and visual elements with full style and layout control. The engine includes a Lua scripting runtime for extending widget functionality with custom logic and data processing, and provides a plugin SDK with a C/C++ API for building native plugins that add new data sources or rendering capa

This is an open-source, crowd-sourced wiki textbook that teaches Linux system programming in C. It covers the core operating system concepts of process management through the fork-exec-wait model, dynamic memory allocation using implicit free list heap allocators, inode-based file systems, inter-process communication via pipes and shared memory, POSIX threads with synchronization primitives, signal-based asynchronous notification, virtual memory with page table translation, and runtime diagnostics using Valgrind and GDB. The textbook distinguishes itself by providing practical, implementation

sds 是一个 C 语言动态字符串库，为堆分配的字符串提供了内存管理封装。它通过显式跟踪字符串长度来实现二进制安全存储，从而允许库处理数据中的空字符。 该库的内存架构支持可互换的分配和释放函数指针，便于集成自定义内存管理器。它利用存储在头部的长度信息实现常数时间的长度获取，并维护以空字符结尾的缓冲区填充，以确保与标准 C 字符串函数的兼容性。 该工具包涵盖了广泛的字符串操作功能，包括拼接、分词、连接和范围提取。它还提供了用于原地缓冲区修改的实用程序（如修剪和裁剪），并包含优于标准打印函数的优化字符串格式化方案。

Globalize 是一个 JavaScript 国际化库和 Unicode CLDR 本地化引擎。它提供了一个工具包，用于使用 Unicode 通用区域数据存储库和 IANA 时区来格式化日期、数字和货币。 该项目具有一个构建时本地化编译器，可将规则预编译为静态格式化程序，以减少运行时开销和包大小。它包含一个使用 ICU MessageFormat 语法解析和格式化复杂字符串的系统，支持变量、复数和性别变体。 该库涵盖了广泛的本地化功能，包括双向日期和时间解析、数字和货币标记化以及相对时间格式化。它管理区域数据加载、全应用语言配置以及用于解析缺失翻译键的消息继承链。

cxxopts 是一个仅包含头文件的 C++ 命令行解析器，用于为控制台应用程序定义必需选项、默认值和位置参数。它作为一个轻量级工具，使用短标志和长标志将命令行参数解析为类型化值，并自动生成格式化的帮助信息和文档。 该库提供了参数规范功能，允许定义选项描述、类型和布尔标志。它支持将多值选项收集到容器中、为省略的选项管理默认值，以及根据顺序将无标志参数映射到特定名称。 该工具包含对选项规范的验证，并能识别无法识别的参数。它在解析阶段通过类型化异常处理错误。

gray-matter 是一个用于分离、解析和序列化文本文档中结构化配置数据的库。它作为一个文档元数据提取器，将结构化块从文件正文中隔离出来，并作为一个序列化库将数据对象转换回格式化字符串。 该项目支持从字符串或文件的开头提取 YAML、JSON 和 TOML 元数据块。它包含识别元数据存在以及根据起始分隔符后的字符确定所用特定格式语言的机制。 该系统通过基于插件的解析引擎和自定义解析器注册进行扩展，允许用户定义逻辑来处理非标准元数据语言。其功能涵盖元数据语言识别、文件元数据提取以及对称序列化以重构原始文档格式。

sqlparse 是一个用于 SQL 语句程序化分析和操作的 Python 库。它提供了一个非验证解析器，将 SQL 文本分解为标记，并提供了一个将包含多个查询的字符串划分为单个语句的实用程序。 该项目包括一个代码格式化程序，使用一致的缩进和关键字大小写来重组 SQL 语句。它还提供了通过版本控制钩子自动化此格式化和 Linting 过程的机制，以保持代码一致性。

TransformerLens is a library for mechanistic interpretability research designed to reverse engineer the learned algorithms within large language models. It provides a standardized framework for wrapping diverse transformer architectures, allowing researchers to extract, manipulate, and analyze internal activations and weights through a consistent interface. The project distinguishes itself through a comprehensive system of activation hooks that can capture, patch, and ablate internal tensors during the forward pass. It includes specialized utilities for decomposing fused projections, material