ZwormZ

> Thanks for getting back to me! > > Regarding the NaNs: I haven't been able to investigate further (no access atm, will do once I fix the other issues)...

Thanks for your quick response! I downloaded your code three times, the first was the original version, the second was on December 6, and the third time was today when...

Thanks for your reply! About the iteration time, the model does run on GPUs, but according to the monitoring, when using deepspeed for multi-card training, the GPUs are occupied but...

Thank you ! I tried move the training dataset to disk and set DataLoader workers to 16. The iteration time did get much shorter, it is now about 60s/it using...

Sure, how can I send it to you conveniently? By email? I will send you the original output loss detail after training 26 epochs, and I'm training again and print...

[loss_1000_6cards.log](https://github.com/aqlaboratory/openfold/files/7843646/loss_1000_6cards.log)

Sorry for the late reply, I spent some time retraining on two V100 cards , here is the loss breakdown. [loss_1000_2cards.log](https://github.com/aqlaboratory/openfold/files/7848852/loss_1000_2cards.log) For visualization, I drew the constituent losses diagram for...

update：after ~4 epoch       

I'd like to test if it's the data that causes the iteration times to be very different in my training process than in your case. It will be great if...