Model constantly collapses, no idea why.

[WelschCode]

I have a pretrain model from scratch LIAE-UD but I've had this issue with LIAE-UDT and also DF-UDT 320 res 384/96/96/32 is the main model I use.. Sometimes I can get it to work all the way through. Most of the time it collapses if I train it for too long though. I do not know why.

The model will work fine for most of training until RW no longer gains details. Then it starts developing either white spots or black holes in the face randomly that morph frame to frame. I have tried different models I've pretrained from scratch and also downloaded other users models with different dims as well.

GPU Temp info (I bought the PC in February) https://ibb.co/VDR9cjf

I've done stress tests on my computer as well and everything is stable as can be.. Checked life span of the m.2 and SSD's I'm using. I've tried training on C:\ I've tried training off of the F:\ I've had. No matter what, it eventually collapses.

Can anyone help me solve this issue?

Example of collapse: https://ibb.co/SfZChpW

[joolstorrentecalo]

Do you have anything like GAN enabled with higher values like >1? Things like this can quickly cause collapse of the model if not kept within normal boundaries.

[joolstorrentecalo]

Issue solved / already answered (or it seems like user error), please close it.