transformers-js icon indicating copy to clipboard operation
transformers-js copied to clipboard

Tokenizer vs SentencePiece: Implementation Similarity and Converting sentencepiece.model to JSON

Open tylike opened this issue 1 year ago • 0 comments

Hi, Is the implementation of tokenizer the same as Google's SentencePiece? For example, will the same input have the same output when calling encode? If so, how can I convert sentencepiece.model file to a json file?

Thank you.

tylike avatar Jul 03 '23 06:07 tylike