Support for russian language

[egorvkid]

Hello! How can i generate questions for russian language?

[mdhasanai]

By using MT5Tokenizer and MT5Model, you can generate question for other language.

[DimIsaev]

@mdhasanai

can you give an example code?

[mdhasanai]

@mdhasanai

can you give an example code?

Use T5Tokenizer when you preprocess the data in prepare_data.py For example, use this

from transformers import MT5Tokenizer, BartTokenizer, HfArgumentParser

instead of

from transformers import T5Tokenizer, BartTokenizer, HfArgumentParser

Replace all the T5Tokenizer/T5Model with MT5Tokenizer/MT5Model

In this way, you can train and evaluate for Non-English dataset. To know more about the MT5 model, follow this link. https://huggingface.co/transformers/model_doc/mt5.html

[DimIsaev]

@mdhasanai

thanks. model and Tokinezir all good

but how prepare datasets ? what data structure should be in the directory where "dev" and "train" are located?