TypeError: LlamaSdpaAttention.forward() got an unexpected keyword argument 'truthx_model'

[GuoqingWang1]

当我从huggingface上下载模型llama2-7b-chat-truthX后 运行text.py时会出现如下错误，请问该如何解决？

[XavierZhang2002]

This issue occurs because the "sdpa" method (LlamaSdpaAttention) is being used, and it seems that TruthX does not support this method. Therefore, you can manually set it to "eager" mode (LlamaAttention) as a workaround.

[OmarMohammed88]

@XavierZhang2002 @GuoqingWang1 ,I got the same error and tried to set the attention to eager mode, but still facing the same issue.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM , AutoConfig

llama2chat_with_truthx = "ICTNLP/Llama-2-7b-chat-TruthX"


# Step 1: Load configuration and override attention implementation
config = AutoConfig.from_pretrained(llama2chat_with_truthx)
config.attn_implementation = "eager"

tokenizer = AutoTokenizer.from_pretrained(llama2chat_with_truthx,)

model = AutoModelForCausalLM.from_pretrained(llama2chat_with_truthx, config=config,
                                             trust_remote_code=True,torch_dtype=torch.float16 ).cuda()

question = "What are the benefits of eating an apple a day?"

encoded_inputs = tokenizer(question, return_tensors="pt")["input_ids"]


outputs = model.generate(encoded_inputs.cuda())[0, encoded_inputs.shape[-1] :]
outputs_text = tokenizer.decode(outputs, skip_special_tokens=True).strip()
print(outputs_text)

TypeError: LlamaSdpaAttention.forward() got an unexpected keyword argument 'truthx_model'