Taku Kudo

Train --model_type=unigram

2

Thank you for making this resource. It would be nice if we can use the model trained with --model_type=unigram, which is a default mode of sentencepiece. BPE and Unigram show...

We are currently using a compatibility subset of the Abseil code, but we would like to replace it with the official package. This is expected to reduce maintenance costs and...

Python module emits the following warnings. We want to suppress them. ``` =================================================================================== test session starts =================================================================================== platform linux -- Python 3.12.11, pytest-8.4.2, pluggy-1.5.0 rootdir: /usr/local/google/home/taku/sentencepiece/python configfile: pyproject.toml collected 1...