for word embedding in RNN model

[Dongximing]

Hi model.embedding.weight.data.copy_ is a fine-tuning way ? so is that mean they are the same as model parameters to train? I also find .pre_train to load word embedding, which means we just use pre-trained embedding and do not change the embedding value (if freeze is true) during training? also, you said to set unk and pad to zero, but why unk is trainable during training? I think they do not change during training since they are zero (zero times all gradient descent value is zero).

[bentrevett]

Yes, if you copy the weights from pre-trained embeddings, then they are fine-tuned. The parameters will update as they are not frozen. If you want to freeze the embedding, then you can set embedding_layer.weight.requires_grad = False.

If using Adam, which we do in the tutorials, then weights initialized to zero will not stay zero due to the beta terms. The weight for the padding token will however stay zero as we pass the padding index to the padding_idx argument of the nn.Embedding layer, which explicitly prevents it from changing.

[Dongximing]

thank u for your kindly reply. I think I got it.