Potential serious precision bug

[JulesGM]

Any reason why https://github.com/artidoro/qlora/blob/cc488110b5ea23594a418daca7085000a9420625/qlora.py#L296 is different from https://github.com/artidoro/qlora/blob/cc488110b5ea23594a418daca7085000a9420625/qlora.py#L279 ?

It gives float32 to torch_dtype in the case of fp16.

[JulesGM]

@artidoro

[artidoro]

Hello @JulesGM we found fp16 to be unstable in our experiments. In the qlora.py script we therefore decided to map to fp32 for the computation datatype when fp16 is specified. Feel free to change the lines and experiment with your own setup. It might not be as big an issue as we encountered initially, however, we preferred to minimize the risk of instabilities in the code we released.