Changing from gpt to gpt2

[vinwinter]

I get an error when I change from gpt to gpt2 in the interactive.py file. However, when I change from gpt to gpt2 in the train.py file, everything is fine.

The error I get: File "/transfer-learning-conv-ai/interact.py", line 130, in run logger.info("Selected personality: %s", tokenizer.decode(chain(*personality)))

File "/pytorch_transformers/tokenization_utils.py", line 767, in decode sub_texts.append(self.convert_tokens_to_string(current_sub_text))

File "/pytorch_transformers/tokenization_gpt2.py", line 198, in convert_tokens_to_string text = ''.join(tokens)

TypeError: sequence item 0: expected str instance, NoneType found

I also tried to switch from pytorch-transformer to the new transformers library to no avail.

[sshleifer]

send full command please

[GraphGrailAi]

@vinwinter This issue was solved somehow or not? I am interested in GPT2 model usage, assuming there is no option to train with gpt2-large but it is interesting how do gpt2 behave. And, assuming GPT2 was released in february 2019 - what is the clear path to use up-to-date data in model (i mean news, wikidata, etc collected after feb 2019) or how to implement OpenGPT2 https://www.reddit.com/r/MachineLearning/comments/cu9xgi/p_opengpt2_we_replicated_gpt2_because_you_can_too/ ?

[ioannist]

@vinwinter This issue was solved somehow or not? I am interested in GPT2 model usage, assuming there is no option to train with gpt2-large but it is interesting how do gpt2 behave. And, assuming GPT2 was released in february 2019 - what is the clear path to use up-to-date data in model (i mean news, wikidata, etc collected after feb 2019) or how to implement OpenGPT2 https://www.reddit.com/r/MachineLearning/comments/cu9xgi/p_opengpt2_we_replicated_gpt2_because_you_can_too/ ?

you can python train.py --model="gpt2-large"

[mriganktiwari]

Has anyone found a solution yet? I tried with 'gpt2-large' as well. And it just bumps into same error:

   **kwargs,
 File "/home/test/miniconda3/envs/qags/lib/python3.6/site-packages/transformers/tokenization_utils.py", line 729, in _decode
   sub_texts.append(self.convert_tokens_to_string(current_sub_text))
 File "/home/test/miniconda3/envs/qags/lib/python3.6/site-packages/transformers/models/gpt2/tokenization_gpt2.py", line 261, in convert_tokens_to_string
   text = "".join(tokens)
TypeError: sequence item 0: expected str instance, NoneType found```

[olegsinavski]

Hello, I had the same problem and found that in my case I was generated token ids out of bounds of a tokenizer. Solution is to remove those:

y = model.generate(...)
y[y >= tokenizer.vocab_size] = tokenizer.unk_token_id