ssszh

@wangzhonghai Hi, have you solved this problem yet? I found the same problem when trying to peft fine-tune CodeLLama-7B (using LlamaForSequenceClassification), the Loss is always Nan during the fine-tuning. Thanks！

@passby111 Hi, have you solved this problem yet? I found the same problem when trying to peft fine-tune CodeLLama-7B (using LlamaForSequenceClassification), the Loss is always 0 during the fine-tuning. Thanks！

> > I found the same problem when trying to peft fine-tune CodeLLama-7B (using LlamaForSequenceClassification), the Loss is always 0 during the fine-tuning. > > Is that also with the...

@BenjaminBossan Thanks for your reply. This is the full code and the data can be downloaded here: https://drive.google.com/drive/folders/1gaZ-pRb07XMMwSnbpBAyjUsm0_08VNrt?usp=drive_link ``` transformers == 4.39.0 peft == 0.6.0 ``` ``` import os os.environ["CUDA_VISIBLE_DEVICES"]...

> @sssszh Thanks for providing the script. I didn't have access to your data, so I requested it. Would this also work with another dataset, like one of the datasets...

> > This dataset is also available on huggingface: `claudios/code_x_glue_devign`, but you may need to modify the code in the Dataset section, because the format of the data is not...

@BenjaminBossan Thank you very much for your reply! I found that the source of the problem may be the use of `torch_dtype=torch.float16`. when using `torch_dtype=torch.float16`, no matter what the configuration...

@sanipanwala Hi, have you solved this problem yet? I found the same problem when trying to peft fine-tune CodeLLama-7B (using LlamaForSequenceClassification), the Loss is always 0 during the fine-tuning. Thanks！

@abhik1505040 Hi, I want to know the result for pass@1 of your fine-tuned model with CE loss for 10 epochs. My result pass@1 It's much lower than the one in...