Traning hangs every 100s of epochs

[claralebbos]

Dear Fabian, thank you for your code. I have a question about the training; I am training a nnUNet model in 5 folds, but every 100s of epochs, my training seems to hang (see attached images). The only workaround this is to continue the training every time it hangs, but this is making the training time much longer. Would you happen to know why this is happening?

Thank you.

[FabianIsensee]

Hi, hard to say because that doesn't happen for me. What configuration are you training? And have you looked at your RAM? If the RAM gets full this can happen Best, Fabian