keras-nlp icon indicating copy to clipboard operation
keras-nlp copied to clipboard

Add a vocab training utility for word piece

Open mattdangerw opened this issue 3 years ago • 1 comments

We would like to add a vocab training utility for wordpiece. This can leverage the utilities for doing this in tensorflow text.

Note that we do not have to cover all the use cases here, or be performant for all data sizes. We should just aim to build a useful tool that covers than majority of common training use cases.

mattdangerw avatar Jul 01 '22 20:07 mattdangerw

@jessechancy will be working on this!

mattdangerw avatar Jul 01 '22 20:07 mattdangerw

This is done!

mattdangerw avatar Oct 18 '23 20:10 mattdangerw