1 مستودع
Awesome GitHub RepositoriesIndentation-Aware Tokenization
Tokenization process that uses character markers and indentation levels to determine structure.
Distinct from String Tokenization: Specializes string tokenization by using indentation state to define block boundaries.
Explore 1 awesome GitHub repository matching software engineering & architecture · Indentation-Aware Tokenization. Refine with filters or upvote what's useful.
Awesome Indentation-Aware Tokenization GitHub Repositories
اعثر على أفضل المستودعات باستخدام الذكاء الاصطناعي.سنبحث عن أفضل المستودعات المطابقة باستخدام الذكاء الاصطناعي.
- nodeca/js-yaml
nodeca/js-yaml
6,591عرض على GitHubjs-yaml is a JavaScript library providing a programmatic interface for parsing and dumping YAML data. It functions as a parser and serializer that converts YAML strings into JavaScript objects and transforms JavaScript objects back into YAML format. The library includes a command-line interface for parsing or dumping YAML data via direct input or data pipes. It also features an abstract syntax tree transformer to modify the structure of data during serialization. The project provides capabilities for multi-document processing and the definition of custom schemas and tags to handle specialize
Breaks input strings into tokens based on indentation and character-level markers.
JavaScript
