open_lm
open_lm copied to clipboard
[WIP] Adding support for FP8 training
- Changes made to
train.py,model.py, andnorms.pyto check if Transformer Engine can be imported for P5 instances or H100s and use FP8 for Linear and LayerNorm layers. - Minor modifications to
main.pyfor FP8 support