1 dépôt
Awesome GitHub RepositoriesTraining
Learns merge rules by iteratively pairing the most frequent adjacent byte sequences in a corpus until a target vocabulary size is reached.
Distinct from Byte-Level Tokenizers: Distinct from Byte-Level Tokenizers: focuses on the training process to learn merge rules, not the application of an existing tokenizer.
Explore 1 awesome GitHub repository matching artificial intelligence & ml · Training. Refine with filters or upvote what's useful.
Awesome Training GitHub Repositories
Trouvez les meilleurs dépôts grâce à l'IA.Nous recherchons les dépôts les plus pertinents grâce à l'IA.
- karpathy/minbpe
karpathy/minbpe
10,582Voir sur GitHubMinimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.
Provides a clean implementation of the BPE training algorithm to learn merge rules from text corpora.
Python
