About the tokenizer

Open yuyq96 opened this issue 3 years ago • 2 comments

Why the tokenizer class specified in tokenizer config is BloomTokenizer? There is only BloomTokenizerFast in transformers implementation.

Apr 19 '23 01:04 yuyq96

Hi @yuyq96,

Thanks for your attention!

Could you upgrade the transformers version: pip install git+https://github.com/huggingface/transformers and try again?

Best, Zhihong

Apr 19 '23 02:04 zhjohnchan

There is no BloomTokenizer even in the latest transformers. https://github.com/huggingface/transformers/tree/main/src/transformers/models/bloom

Apr 19 '23 07:04 yuyq96

Hi yuyq96,

I got the same issue. Got any solution? Thanks in advance.

There is no BloomTokenizer even in the latest transformers. https://github.com/huggingface/transformers/tree/main/src/transformers/models/bloom

May 01 '23 04:05 dickchanym

@dickchanym I manually switched to BloomTokenizerFast and it works fine.

May 04 '23 08:05 yuyq96