# xujiajun/gotokenizer

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/xujiajun-gotokenizer).**

21 stars · 7 forks · Go · Apache-2.0

## Links

- GitHub: https://github.com/xujiajun/gotokenizer
- awesome-repositories: https://awesome-repositories.com/repository/xujiajun-gotokenizer.md

## Description

A tokenizer based on the dictionary and Bigram language models for Go. (Now only support chinese segmentation)

## Tags

### Part of an Awesome List

- [Text Tokenization](https://awesome-repositories.com/f/awesome-lists/ai/text-tokenization.md) — Tokenizer based on dictionary and Bigram language models.