# arbox/tokenizer

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/arbox-tokenizer).**

0 stars · 0 forks

## Links

- GitHub: https://github.com/arbox/tokenizer
- awesome-repositories: https://awesome-repositories.com/repository/arbox-tokenizer.md

## Tags

### Part of an Awesome List

- [Text Tokenization and Segmentation](https://awesome-repositories.com/f/awesome-lists/devtools/text-tokenization-and-segmentation.md) — Simple multilingual text tokenizer.