# kaleidophon/token2index

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/kaleidophon-token2index).**

50 stars · 6 forks · Python · GPL-3.0

## Links

- GitHub: https://github.com/Kaleidophon/token2index
- Homepage: http://token2index.readthedocs.io
- awesome-repositories: https://awesome-repositories.com/repository/kaleidophon-token2index.md

## Topics

`deep-learning` `deeplearning` `i2t` `i2w` `indexing` `itos` `nlp` `numpy` `python` `pytorch` `rnn` `rnns` `seq2seq` `stoi` `t2i` `tensorflow` `token` `transformer` `transformers` `w2i`

## Description

A lightweight but powerful library to build token indices for NLP tasks, compatible with major Deep Learning frameworks like PyTorch and Tensorflow.

## Tags

### Part of an Awesome List

- [Natural Language Processing](https://awesome-repositories.com/f/awesome-lists/ai/natural-language-processing.md) — Listed in the “Natural Language Processing” section of the FunNLP awesome list.
- [Lexical Analysis Tools](https://awesome-repositories.com/f/awesome-lists/devtools/lexical-analysis-tools.md) — Lightweight library for mapping tokens to indices.