AI search

Explore more awesome repositories

Describe what you need in plain English — the AI ranks thousands of curated open-source projects by relevance.

Tokenizer Persisters - Ships built-in serialization for saving and loading trained tokenizer models to disk.

Byte Level Encodings - Encodes UTF-8 text into byte-level token sequences using BPE merges for subword tokenization.

Byte Pair Encodings - Provides a clean implementation of the BPE algorithm used for subword tokenization in large language models.

Greedy Merge Encoding - Encodes text by greedily applying learned byte pair merges until no more merges apply.

Training - Trains a BPE tokenizer on text data to learn merge rules and build a vocabulary for subword tokenization.

Byte-Level Vocabulary Trainers - Trains byte-level BPE vocabularies and merge rules on text corpora for subword tokenization.

Text Tokenization - Converts raw text into integer token IDs using a trained BPE tokenizer for language model pipelines.

Byte-Level Tokenizers - Implements a byte-level tokenizer that maps integer IDs to raw byte sequences for full UTF-8 coverage.

Training - Provides a clean implementation of the BPE training algorithm to learn merge rules from text corpora.

Subword Tokenization - Provides a framework for splitting text into subword units with support for special tokens and regex pre-tokenization.

Byte-Level Encoders - Encodes raw text into integer token IDs using byte-level BPE merges for language model pipelines.

Token Decoders - Provides a decoder that reconstructs human-readable text from integer token IDs using the trained vocabulary.

Greedy Merge Application - Implements greedy merge application that repeatedly applies the highest-priority learned byte pair merge.

Merge Tables - Ships a priority-ordered merge table that stores learned byte pair rules for sequential application.

Category-Based Splitters - Splits text by character category before tokenization to prevent cross-category BPE merges.

Persistence - Saves and loads a trained BPE tokenizer's vocabulary and merge rules to disk for reuse across sessions.

Serialization - Provides a module for saving and loading trained tokenizer models with vocabulary and merge tables to disk.

Category-Based Splitters - Implements regex-based text splitting by category to prevent cross-category BPE merges during tokenization.

Pre-Tokenization Splitters - Ships a regex-based pre-splitter that segments text into categories before BPE merges.

Regex Column Splitting - Splits input text into categories like letters, numbers, and punctuation using a configurable regex pattern.

Special Token Registries - Provides a mechanism to register custom tokens with reserved IDs that are never split or merged.

Tokenizer State Persistence - Saves and restores the full tokenizer state including vocabulary, merge table, and special tokens to disk.

Minbpe