sentencepiece

Unsupervised text tokenizer and detokenizer

Description

SentencePiece is an unsupervised text tokenizer/detokenizer mainly designed for Neural Network-based text generation systems where the vocabulary size is predetermined prior to the neural model training.

Upload more screenshots

Please help extend the collection of screenshots. Just make a screenshot and upload it here. You don't need to register or anything.

Upload a screenshot

Hint: upload an image here from your clipboard with Ctrl-V


Homepage

https://github.com/google/sentencepiece


Install this software package

If the package is available for the distribution you are currently using on your computer then install the software by clicking on…

Install sentencepiece