awesome-repositories.comBlog

© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com

MCP Blog Curated searches Sitemap Privacy Terms

Gotokenizer | Awesome Repository

xujiajungotokenizer

0

View on GitHub↗

21 stars·7 forks·Go·Apache-2.0·0 views

Gotokenizer

Features

Text Tokenization - Tokenizer based on dictionary and Bigram language models.

AI search

Explore more awesome repositories

Describe what you need in plain English — the AI ranks thousands of curated open-source projects by relevance.

Start searching with AI

A tokenizer based on the dictionary and Bigram language models for Go. (Now only support chinese segmentation)