1 repository
Awesome GitHub RepositoriesCategory-Based Splitters
Splits input text by category (letters, numbers, punctuation) using a regex pattern before tokenization to prevent cross-category merges.
Distinct from Text Preprocessing: Distinct from Text Preprocessing: specifically splits text by character category to prevent BPE merges across category boundaries, not general cleaning or normalization.
Explore 1 awesome GitHub repository matching data & databases · Category-Based Splitters. Refine with filters or upvote what's useful.
Awesome Category-Based Splitters GitHub Repositories
Găsește cele mai bune repo-uri cu AI.Vom căuta cele mai potrivite repository-uri folosind AI.
- karpathy/minbpe
karpathy/minbpe
10,582Vezi pe GitHubMinimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.
Implements regex-based text splitting by category to prevent cross-category BPE merges during tokenization.
Python
