Adapters cannot be correctly loaded using PeftModel.from_pretrained()

[HenryCai11]

System Info

peft==0.10.0 transformers=4.39.3

Who can help?

No response

Information

• [ ] The official example scripts
• [X] My own modified scripts

Tasks

• [ ] An officially supported task in the examples folder
• [X] My own task or dataset (give details below)

Reproduction

from peft import PeftModel, AdaptionPromptConfig
from safetensors import safe_open

# The adapter I'm using
config = AdaptionPromptConfig(
    adapter_len=20,
    adapter_layers=32,
    task_type="CAUSAL_LM",
    target_modules="self_attn",
)

model_name_or_path = "meta-llama/Llama-2-7b-chat-hf"
model = AutoModelForCausalLM.from_pretrained(model_name_or_path, low_cpu_mem_usage=True, torch_dtype=torch.float32, device_map="cuda:1", trust_remote_code=True).eval()
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, padding_side='left')
tokenizer.pad_token_id = 0
tokenizer.bos_token_id = 1

peft_model = PeftModel.from_pretrained(model, "HenryCai1129/adapter-emo")

tensors = {}
with safe_open("{downloaded_lora_dir}", framework="pt", device=1) as f:
    for k in f.keys():
        tensors[k] = f.get_tensor(k)

Expected behavior

The tensors should be identical to the lora parameters in the loaded peft_model. However, I got different parameters and the adaption_gates are all zero, which may indicate that it is an randomly-initialized adaption prompt.

[HenryCai11]

Additional Info: It turns out that the issue is kinda solved when I switch to another machine. I'm not sure if the problem comes from server-side (say, cuda version, etc.) or from dependencies. I tried different versions of transformers and peft, sometimes it worked, but most of the time it did not.

[BenjaminBossan]

Some questions: Your text mentions LoRA but your config is AdaptionPromptConfig, same for the linked adapter, so which is it? Also, could you please show how you compare the weights?

It turns out that the issue is kinda solved when I switch to another machine. I'm not sure if the problem comes from server-side (say, cuda version, etc.) or from dependencies. I tried different versions of transformers and peft, sometimes it worked, but most of the time it did not.

Could you give more details? What do you mean by "kinda solved"? Did you run the exact same code and it worked on another machine? What are the specs and package versions on those machines?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.