Resize position embeddings in PreTrainedModel

[PaulLerner]

Feature request

Add a method to resize position embeddings in PreTrainedModel, in the same way as there is resize_token_embeddings for word embeddings.

There are several ways to do that:

• retrain everything from scratch
• keep the pretrained embeddings but add new trained from scratch for the new positions (as done in PreTrainedModel._get_resized_embeddings if I understand correctly)
• same but initialize new positions by interpolating pretrained ones instead of random init

Motivation

It would be nice to be able to resize position embeddings when the PreTrainedModel has too small max_position_embeddings. I found several related issues:

• https://stackoverflow.com/questions/69820065/how-to-extend-a-pretrained-transformer-model-configured-with-small-max-position
• https://github.com/huggingface/transformers/issues/1978

Your contribution

Willing to help :) From what I can tell, most of the job is already done in PreTrainedModel._get_resized_embeddings

[jack-gits]

I have met the same issue.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[rhnfzl]

I am also facing the same issue

[lyriccoder]

is this issue resolved? Why was it closed? I need to use different size, not, but I can't do it

[PaulLerner]

It was closed automatically because no one answered after one month :man_shrugging:

[ImmortalSdm]

They exactly a PR for the same issue, but only for some models. #13559 I think you can modify the resize_position_embeddings in your own model based on the example in this PR.