1 个仓库
Awesome GitHub RepositoriesCategory-Based Splitters
Splitting text by category (letters, numbers, punctuation) before tokenization to prevent cross-category merges.
Distinct from Recursive Text Splitting: Distinct from Recursive Text Splitting: splits by character category for BPE preprocessing, not recursive chunking for token limits.
Explore 1 awesome GitHub repository matching artificial intelligence & ml · Category-Based Splitters. Refine with filters or upvote what's useful.
Awesome Category-Based Splitters GitHub Repositories
用 AI 发现最棒的仓库。我们将通过 AI 为您搜索最匹配的仓库。
- karpathy/minbpe
karpathy/minbpe
10,582在 GitHub 上查看Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.
Splits text by character category before tokenization to prevent cross-category BPE merges.
Python
