Hailin Luo

> 我在我们的环境中测试，无论单卡还是多卡都没有遇到这个问题，请问大家在自己的训练中能够稳定复现这个问题吗？ > > We didn't encounter this issue in our environtment, whether with a single GPU or multiple GPUs. Can you consistently reproduce this problem during your training? 在3090上，单卡和多卡都稳定复现，只能注释掉evaluate的代码

> > > 我在我们的环境中测试，无论单卡还是多卡都没有遇到这个问题，请问大家在自己的训练中能够稳定复现这个问题吗？ > > > We didn't encounter this issue in our environtment, whether with a single GPU or multiple GPUs. Can you consistently reproduce this problem during...

一个简单的解决办法就是：在eval的时候，传给log_validatation的net，做一个深拷贝就好了，避免覆盖训练的net 把 ``` reference_unet = ori_net.reference_unet denoising_unet = ori_net.denoising_unet ``` 改成 ``` reference_unet = copy.deepcopy(ori_net.reference_unet) denoising_unet = copy.deepcopy(ori_net.denoising_unet) ```