keras-nlp icon indicating copy to clipboard operation
keras-nlp copied to clipboard

Add a model proto training utility for sentence piece

Open mattdangerw opened this issue 3 years ago • 2 comments

Along with https://github.com/keras-team/keras-nlp/issues/248, we should add a utility to train a model proto for sentence piece. This can leverage the sentencepiece pip package to run their trainer.

We should not expose most of the option the sentencepiece trainer has, the goal will be for this to cover the majority of use cases, with advanced users till using the sentencepiece pip package directly.

mattdangerw avatar Jul 01 '22 20:07 mattdangerw

@aflah02 here is the sentencepiece one!

mattdangerw avatar Jul 01 '22 20:07 mattdangerw

Thanks @mattdangerw 😁

aflah02 avatar Jul 01 '22 20:07 aflah02