Add an option to predefine special tokens

[Kyeongpil]

I want to use this yttm model.

However, I want to add [MASK] token to the vocabulary.

In this case, How can I predefine special tokens?

[xbelonogov]

We don't support this feature right now. Maybe it will be added later.

You can use the following workaround. Just append this special token in the end of your training text many times (for example 1000). In this case the model will definitely add it to vocabulary.

[Kyeongpil]

@xbelonogov Thank you for your kind response.

I'll be looking forward this feature to be added. Because many language models such as BERT needs special tokens such as cls, sep, mask.

[glample]

We would also find this option very useful 👍 The workaround to have 1000 special tokens at the beginning is fine if there are just a few tokens, but is inconvenient when we want to provide a large set of words that should never be split.

[miguelvictor]

Any progress on this ? 😊