1 repository
Awesome GitHub RepositoriesTraining
Learns merge rules by iteratively pairing the most frequent adjacent byte sequences in a corpus until a target vocabulary size is reached.
Distinct from Byte-Level Tokenizers: Distinct from Byte-Level Tokenizers: focuses on the training process to learn merge rules, not the application of an existing tokenizer.
Explore 1 awesome GitHub repository matching artificial intelligence & ml · Training. Refine with filters or upvote what's useful.
Awesome Training GitHub Repositories
Găsește cele mai bune repo-uri cu AI.Vom căuta cele mai potrivite repository-uri folosind AI.
- karpathy/minbpe
karpathy/minbpe
10,582Vezi pe GitHubMinimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.
Provides a clean implementation of the BPE training algorithm to learn merge rules from text corpora.
Python
