Supporting LmHead and Embedding Layers for Adapters

[magdyksaleh]

System Info

Doesn't work if you make changes to the vocab

Information

• [ ] Docker
• [ ] The CLI directly

Tasks

• [ ] An officially supported command
• [ ] My own modifications

Reproduction

to come

Expected behavior

to come

[tgaddair]

Context: https://stackoverflow.com/questions/72775559/resize-token-embeddings-on-the-a-pertrained-model-with-different-embedding-size

[arnavgarg1]

Here's a code block also demonstrating what might be needed:

>>> from transformers import AutoTokenizer, AutoModelForCausalLM
>>> model = AutoModelForCausalLM.from_pretrained("yujiepan/llama-2-tiny-random")
>>> tokenizer = AutoTokenizer.from_pretrained("yujiepan/llama-2-tiny-random")
>>> model.get_input_embeddings()
Embedding(32000, 8, padding_idx=0)
>>> len(tokenizer.vocab)
32000
>>> tokenizer.add_tokens(['|INST|'])
1
>>> len(tokenizer.vocab)
32001
>>> model.resize_token_embeddings(len(tokenizer.vocab))
Embedding(32001, 8)
>>> model.get_input_embeddings().padding_idx = 0 # Save before and set again after resizing
>>> model.get_input_embeddings()
Embedding(32001, 8, padding_idx=0)