8-bit precision training

[rasbt]

trafficstars

Might be worthile to look into

from torchao.float8 import convert_to_float8_training
convert_to_float8_training(model)

some time

[mtasic85]

I watched this week this video: https://www.youtube.com/watch?v=Br07GsnnvWc

This performance is expected:

About Embedding layer, it should be investigated if they use appropriate class like this one, or we need to take care of it: https://github.com/pytorch/ao/blob/e7b33bc91c831d10249c1222c8b4b667f18f28b7/torchao/quantization/prototype/qat/embedding.py#L27

[rasbt]

Thanks for sharing this. Need to check out the video some time. It's long (but good explanations of tricky things are naturally long) but looks really informative.

[mtasic85]

Hi @t-vi, I see that you are quite active here. Any chance to get this integrated in litgpt?

[t-vi]

Thank you for pinging @mtasic85 . We're looking into more fp8 support, but we likely want to deliver this through Thunder, which will compile models to use optimizations. We do have initial fp8 support through NVIDIA TransformerEngine, starting with matmuls. We will likely show examples in ~1 month to 6 weeks.

[codestar12]

any update on this?