# huggingface/tokenizers

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/huggingface-tokenizers).**

10,825 stars · 1,127 forks · Rust · Apache-2.0

## Links

- GitHub: https://github.com/huggingface/tokenizers
- Homepage: https://huggingface.co/docs/tokenizers
- awesome-repositories: https://awesome-repositories.com/repository/huggingface-tokenizers.md

## Topics

`bert` `gpt` `language-model` `natural-language-processing` `natural-language-understanding` `nlp` `transformers`

## Description

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

## Tags

### Part of an Awesome List

- [Artificial Intelligence](https://awesome-repositories.com/f/awesome-lists/ai/artificial-intelligence.md) — Modern NLP tokenization pipelines with high-performance implementations.
- [Model Serving Engines](https://awesome-repositories.com/f/awesome-lists/ai/model-serving-engines.md) — High-performance tokenization library for research and production.
- [Natural Language Processing](https://awesome-repositories.com/f/awesome-lists/ai/natural-language-processing.md) — High-performance tokenization library optimized for research and production.
- [Lexical Analysis Tools](https://awesome-repositories.com/f/awesome-lists/devtools/lexical-analysis-tools.md) — High-performance tokenization library for NLP models.