open_lm icon indicating copy to clipboard operation
open_lm copied to clipboard

[WIP] Adding support for FP8 training

Open shahromil16 opened this issue 1 year ago • 0 comments

  • Changes made to train.py, model.py, and norms.py to check if Transformer Engine can be imported for P5 instances or H100s and use FP8 for Linear and LayerNorm layers.
  • Minor modifications to main.py for FP8 support

shahromil16 avatar Feb 21 '24 20:02 shahromil16