1 مستودع
Awesome GitHub RepositoriesTraining
Learns merge rules by iteratively pairing the most frequent adjacent byte sequences in a corpus until a target vocabulary size is reached.
Distinct from Byte-Level Tokenizers: Distinct from Byte-Level Tokenizers: focuses on the training process to learn merge rules, not the application of an existing tokenizer.
Explore 1 awesome GitHub repository matching artificial intelligence & ml · Training. Refine with filters or upvote what's useful.
Awesome Training GitHub Repositories
اعثر على أفضل المستودعات باستخدام الذكاء الاصطناعي.سنبحث عن أفضل المستودعات المطابقة باستخدام الذكاء الاصطناعي.
- karpathy/minbpe
karpathy/minbpe
10,582عرض على GitHubMinimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.
Provides a clean implementation of the BPE training algorithm to learn merge rules from text corpora.
Python
