1 repo
Awesome GitHub RepositoriesTokenization Customizers
Tools for overriding how text is identified and converted into tokens.
Distinguishing note: Focuses on the tokenization logic specifically.
Explore 1 awesome GitHub repository matching content management & publishing · Tokenization Customizers. Refine with filters or upvote what's useful.
Awesome Tokenization Customizers GitHub Repositories
Find the best repos with AI.We'll search the best matching repositories with AI.
- markedjs/marked
markedjs/marked
36,608View on GitHubThis project is a high-performance markdown-to-HTML parser designed for use in browser, server-side, and command-line environments. It functions as a configurable syntax processor that transforms plain text documents into structured web content, providing a flexible engine for rendering dynamic documentation and web-based text. The parser features a modular, extensible pipeline that allows developers to intercept the document transformation process at multiple stages. Through custom tokenization, rendering overrides, and lifecycle hooks, users can define unique syntax, modify the token stream
Defines custom logic for identifying and converting specific markdown syntax into tokens.
TypeScriptcommonmarkcompilergfm
