1 مستودع
Awesome GitHub RepositoriesCategory-Based Splitters
Splits input text by category (letters, numbers, punctuation) using a regex pattern before tokenization to prevent cross-category merges.
Distinct from Text Preprocessing: Distinct from Text Preprocessing: specifically splits text by character category to prevent BPE merges across category boundaries, not general cleaning or normalization.
Explore 1 awesome GitHub repository matching data & databases · Category-Based Splitters. Refine with filters or upvote what's useful.
Awesome Category-Based Splitters GitHub Repositories
اعثر على أفضل المستودعات باستخدام الذكاء الاصطناعي.سنبحث عن أفضل المستودعات المطابقة باستخدام الذكاء الاصطناعي.
- karpathy/minbpe
karpathy/minbpe
10,582عرض على GitHubMinimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.
Implements regex-based text splitting by category to prevent cross-category BPE merges during tokenization.
Python
