1 Repo
Awesome GitHub RepositoriesCategory-Based Splitters
Splits input text by category (letters, numbers, punctuation) using a regex pattern before tokenization to prevent cross-category merges.
Distinct from Text Preprocessing: Distinct from Text Preprocessing: specifically splits text by character category to prevent BPE merges across category boundaries, not general cleaning or normalization.
Explore 1 awesome GitHub repository matching data & databases · Category-Based Splitters. Refine with filters or upvote what's useful.
Awesome Category-Based Splitters GitHub Repositories
Finde die besten Repos mit KI.Wir suchen mit KI nach den am besten passenden Repositories.
- karpathy/minbpe
karpathy/minbpe
10,582Auf GitHub ansehenMinimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.
Implements regex-based text splitting by category to prevent cross-category BPE merges during tokenization.
Python
