Use Roberta in pytorch transformers

[Vimos]

I have been using BertPreTrainedModel to load this roberta model, which works well.

Noticing in pytorch_transformers, Roberta is also supported.

from pytorch_transformers import (BertConfig, BertTokenizer,
                                  RobertaConfig, RobertaTokenizer)

Should I switch to Roberta? If so, what to use for the parameter merges_file in RobertaTokenizer?

[brightmart]

no. this roberta_zh is changed based on bert with main ideas from RoBERTa paper, but it may not compatible to roberta pytorch implementation from facebook ai project. so load from BertPreTrainedModel should be right way.

can you paste code here to show how you load roberta_zh using BertPreTrainedModel, and how you use it.

(anyway you can have a try, but it should be failed)

[Vimos]

It's the same with original pytorch-transformers

bert_config = MODEL_CLASSES[args.model_type][0].from_json_file(args.bert_config_file)
tokenizer = MODEL_CLASSES[args.model_type][2](vocab_file=args.vocab_file, 
do_lower_case=args.do_lower_case)
model = MODEL_CLASSES[args.model_type][1].from_pretrained(args.model_name_or_path,
                from_tf=bool('.ckpt' in args.model_name_or_path), config=bert_config)

However, for RobertaTokenizer, you cannot use it to load roberta_zh.

• It requires merges_file parameter which is not in roberta_zh

Traceback (most recent call last):
  File "run_transformers.py", line 447, in <module>
    main()
  File "run_transformers.py", line 389, in main
    tokenizer = MODEL_CLASSES[args.model_type][2](vocab_file=args.vocab_file, do_lower_case=args.do_lower_case)
TypeError: __init__() missing 1 required positional argument: 'merges_file'

• If you use from_pretrained, it complains an encoding problem of roberta_zh.

09/28/2019 14:23:53 - INFO - pytorch_transformers.tokenization_utils -   Model name '/home/vimos/.pytorch_pretrained_bert/roberta_zh/pytorch_model.bin' nsubsubsectionot found in model shortcut name list (roberta-base, roberta-large, roberta-large-mnli). Assuming '/home/vimos/.pytorch_pretrained_bert/roberta_zh/pytorch_model.bin' is a path or url to a directory containing tokenizer files.
09/28/2019 14:23:53 - INFO - pytorch_transformers.tokenization_utils -   Didn't find file /home/vimos/.pytorch_pretrained_bert/roberta_zh/added_tokens.json. We won't load it.
09/28/2019 14:23:53 - INFO - pytorch_transformers.tokenization_utils -   Didn't find file /home/vimos/.pytorch_pretrained_bert/roberta_zh/special_tokens_map.json. We won't load it.
09/28/2019 14:23:53 - INFO - pytorch_transformers.tokenization_utils -   Didn't find file /home/vimos/.pytorch_pretrained_bert/roberta_zh/tokenizer_config.json. We won't load it.
09/28/2019 14:23:53 - INFO - pytorch_transformers.tokenization_utils -   loading file /home/vimos/.pytorch_pretrained_bert/roberta_zh/pytorch_model.bin
09/28/2019 14:23:53 - INFO - pytorch_transformers.tokenization_utils -   loading file /home/vimos/.pytorch_pretrained_bert/roberta_zh/pytorch_model.bin
09/28/2019 14:23:53 - INFO - pytorch_transformers.tokenization_utils -   loading file None
09/28/2019 14:23:53 - INFO - pytorch_transformers.tokenization_utils -   loading file None
09/28/2019 14:23:53 - INFO - pytorch_transformers.tokenization_utils -   loading file None
Traceback (most recent call last):
  File "run_transformers.py", line 448, in <module>
    main()
  File "run_transformers.py", line 389, in main
    tokenizer = MODEL_CLASSES['roberta'][2].from_pretrained(args.model_name_or_path)
  File "/home/vimos/anaconda3/envs/rcqa/lib/python3.7/site-packages/pytorch_transformers/tokenization_utils.py", line 293, in from_pretrained
    return cls._from_pretrained(*inputs, **kwargs)
  File "/home/vimos/anaconda3/envs/rcqa/lib/python3.7/site-packages/pytorch_transformers/tokenization_utils.py", line 421, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/home/vimos/anaconda3/envs/rcqa/lib/python3.7/site-packages/pytorch_transformers/tokenization_roberta.py", line 82, in __init__
    mask_token=mask_token, **kwargs)
  File "/home/vimos/anaconda3/envs/rcqa/lib/python3.7/site-packages/pytorch_transformers/tokenization_gpt2.py", line 118, in __init__
    self.encoder = json.load(open(vocab_file, encoding="utf-8"))
  File "/home/vimos/anaconda3/envs/rcqa/lib/python3.7/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/home/vimos/anaconda3/envs/rcqa/lib/python3.7/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

[brightmart]

yeah, it is.

[mayktian]

can you please suggest which pytorch bert package is compatible with this pre-trained model? I had the same failure as Vimos when loading it with facebook transformers.

[brightmart]

roberta is bert,you can load roberta use pytorch transformer, select model as bert