Shu Wang comments

Results 9 comments of


                                            Shu Wang

Change License to LGPL

OK, I will change the license.

Add chat mode for finetuned models using Adapter.

This is a separate script added to the ./chat/ folder, won't affect the other script logic.

probability tensor contains either `inf`, `nan` or element < 0

> How about your loss values? Do they become nan after step 1? I believe this might be related to the loss nan problem.

probability tensor contains either `inf`, `nan` or element < 0

Solved when I changed to A100

probability tensor contains either `inf`, `nan` or element < 0

Yes.

Loss is NaN when finetuning with FSDP

Same issue. Any suggestions on the finetune learning rate or any settings to fix it? iter 0 step 0: loss 1.7089, train time: 1409.17ms iter 1 step 0: loss 2.1155,...

Loss is NaN when finetuning with FSDP

I still have this problem using the newest source code. I am using INCITE-3b with dolly (taking about 8+GB) in 1 GPU (2080Ti) and 16-true precision.

Finetune Falcon-40B with adapter_v2.py using 8 A100 80GB GPUs

I will try, which hyper-parameters are you using?

Security patch dataset inaccessible.

It might exceed the access window. You can forward this issue to [email protected] with your previous application email. Once this issue has been solved, please download the dataset as soon...