1 repositorio
Awesome GitHub RepositoriesTraining
Learns merge rules by iteratively pairing the most frequent adjacent byte sequences in a corpus until a target vocabulary size is reached.
Distinct from Byte-Level Tokenizers: Distinct from Byte-Level Tokenizers: focuses on the training process to learn merge rules, not the application of an existing tokenizer.
Explore 1 awesome GitHub repository matching artificial intelligence & ml · Training. Refine with filters or upvote what's useful.
Awesome Training GitHub Repositories
Encuentra los mejores repositorios con IA.Buscaremos los repositorios que mejor coincidan usando IA.
- karpathy/minbpe
karpathy/minbpe
10,582Ver en GitHubMinimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.
Provides a clean implementation of the BPE training algorithm to learn merge rules from text corpora.
Python
