pytorch-openai-transformer-lm icon indicating copy to clipboard operation
pytorch-openai-transformer-lm copied to clipboard

Published 20 hours ago • verified publisher icon huggingface

So we can not change the word embedding with the pretrained LM?

Open herbertchen1 opened this issue 6 years ago • 1 comments

And training the LM is very hard...?

herbertchen1 avatar Aug 23 '18 04:08 herbertchen1

I do not think that you can change the word embedding easily since its dimension must be the same as the output of each layer, in the case of the pre-trained model 768 (cfg.n_embd).

Training a new language model from scratch is indeed quite expensive and tedious.

rodgzilla avatar Aug 23 '18 10:08 rodgzilla