karpathy/minbpe
Minbpe
Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.
Features
- Toolkits - Provides a complete toolkit for learning BPE merge rules and building a vocabulary from text corpora.
- Tokenizer Persisters - Ships built-in serialization for saving and loading trained tokenizer models to disk.
- Byte Level Encodings - Encodes UTF-8 text into byte-level token sequences using BPE merges for subword tokenization.
- Byte Pair Encodings - Provides a clean implementation of the BPE algorithm used for subword tokenization in large language models.
- Greedy Merge Encoding - Encodes text by greedily applying learned byte pair merges until no more merges apply.
- Training - Trains a BPE tokenizer on text data to learn merge rules and build a vocabulary for subword tokenization.
- Byte-Level Vocabulary Trainers - Trains byte-level BPE vocabularies and merge rules on text corpora for subword tokenization.
- Text Tokenization - Converts raw text into integer token IDs using a trained BPE tokenizer for language model pipelines.
- Byte-Level Tokenizers - Implements a byte-level tokenizer that maps integer IDs to raw byte sequences for full UTF-8 coverage.
- Training - Provides a clean implementation of the BPE training algorithm to learn merge rules from text corpora.
- Subword Tokenization - Provides a framework for splitting text into subword units with support for special tokens and regex pre-tokenization.
- Byte-Level Encoders - Encodes raw text into integer token IDs using byte-level BPE merges for language model pipelines.
- Token Decoders - Provides a decoder that reconstructs human-readable text from integer token IDs using the trained vocabulary.
- Greedy Merge Application - Implements greedy merge application that repeatedly applies the highest-priority learned byte pair merge.
- Merge Tables - Ships a priority-ordered merge table that stores learned byte pair rules for sequential application.
- Category-Based Splitters - Splits text by character category before tokenization to prevent cross-category BPE merges.
- Persistence - Saves and loads a trained BPE tokenizer's vocabulary and merge rules to disk for reuse across sessions.
- Serialization - Provides a module for saving and loading trained tokenizer models with vocabulary and merge tables to disk.
- Category-Based Splitters - Implements regex-based text splitting by category to prevent cross-category BPE merges during tokenization.
- Pre-Tokenization Splitters - Ships a regex-based pre-splitter that segments text into categories before BPE merges.
- Regex Column Splitting - Splits input text into categories like letters, numbers, and punctuation using a configurable regex pattern.
- Special Token Registries - Provides a mechanism to register custom tokens with reserved IDs that are never split or merged.
- Tokenizer State Persistence - Saves and restores the full tokenizer state including vocabulary, merge table, and special tokens to disk.
