# proycon/python-ucto

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/proycon-python-ucto).**

31 stars · 5 forks · Cython

## Links

- GitHub: https://github.com/proycon/python-ucto
- awesome-repositories: https://awesome-repositories.com/repository/proycon-python-ucto.md

## Description

This is a Python binding to the tokenizer Ucto. Tokenisation is one of the first step in almost any Natural Language Processing task, yet it is not always as trivial a task as it appears to be. This binding makes the power of the ucto tokeniser available to Python. Ucto itself is regular-expression based, extensible, and advanced tokeniser written in C++ (http://ilk.uvt.nl/ucto).

## Tags

### Part of an Awesome List

- [Natural Language Processing](https://awesome-repositories.com/f/awesome-lists/ai/natural-language-processing.md) — Unicode-aware rule-based tokenizer for multiple languages.
- [Python NLP Libraries](https://awesome-repositories.com/f/awesome-lists/devtools/python-nlp-libraries.md) — Unicode-aware tokenizer with support for various languages.