pretrain.py of llama-7b model, Exception: : Current loss scale already at minimum

[liukaiyueyuo]

when I run pretrain.py of llama-7b model, it has exception below:

ExceptionException: : Current loss scale already at minimum - cannot decrease scale anymore. Exiting run.Current loss scale already at minimum - cannot decrease scale anymore. Exiting run.

what's problem? How to solve?

[liukaiyueyuo]

My GPU server：
GPU：single A100（80G） memory： 128G

[Abolfazl-kr]

i think it's good to try bf16. it will work with A100 and the problem of loss scale