attn impl to sdpa...

[saa1028]

new version of transfomer, no need to use BetterTransformer, try setting attn impl to sdpa... attn imp: <class 'transformers.models.llama.modeling_llama.LlamaSdpaAttention'>

[profintegra]

I have the same issue

[profintegra]

Solution: in your python code, insert line: model.tokenizer.pad_token = model.tokenizer.eos_token before this line: input_tokens = model.tokenizer(input_text, ......

[ahmedbr]

I have same problem. Any updates on this?

[leedahae340]

这个不是问题，和这里有关系max_new_tokens=20，如果是20，就要跑20次，如果是200，就要跑200次。。。 有点慢