Purvang

2023-05-01 15:16:45 | INFO | yolox.core.trainer:259 - epoch: 2/300, iter: 250/313, mem: 11174Mb, iter_time: 3.676s, data_time: 3.100s, total_loss: 8.0, iou_loss: 3.3, l1_loss: 0.0, conf_loss: 3.4, cls_loss: 0.9, seg_loss: 0.3, lr:...

[BUG]: loading OPT 66B model - CPU runs out of memory

2

### Is there an existing issue for this bug? - [X] I have searched the existing issues ### 🐛 Describe the bug I am trying to reproduce OPT-66B using 16xH100...

**Describe the bug** I am trying to convert the [OPT-350m](https://huggingface.co/facebook/opt-350m) hugging face model to Nemo format. How can I map the following keys? ``` - model.decoder.project_out.weight - model.decoder.project_in.weight ``` for...

SFT training getting nan loss when using PP=4, TP=4 and model params > 7b

2

**Describe the bug** I am trying to finetune the OPT-13b model and getting nan loss at step=4 with following configuration. PP=4 TP=4 MBS=4 Batch size=128 Running model less than or...