Fisher

@Amelia0911 你好，我同样用twoPositionPTZCam中的700多张图像做训练，源码基本没改动，但是模型好像没有学习的趋势，changemaps都是一片蓝，loss也没有下降的趋势，请问你对代码有做什么改动吗，label是用gtbinary吗 Edit: 自己改了改模型和超参数，可以训练了

@auvipy what's your recommendation about version?

I think this may be because you have not saved the optimizer parameters

@JiaminRen my code is stuck at dist.broadcast, no error message, backend is nccl. Do you test the train code or what configuration I didn’t do

@JiaminRen I just tried to imitate your train code in face recognition to use SyncSN in my code, but I didn't succeed. I met two problem, frist, rank = int(os.environ['RANK'])...

@JiaminRen I quickly test the face_recognition train.py, unfortunately I met the same problems. I think maybe some system configurations I missed.

@JiaminRen my system is ubuntu18.04 and I use ananconda to install pytorch, program is stuck at dist.broadcast

Thanks, `torch.distributed.launch` can solve problems. But sync way consumes a lot of GPU memory, always out of memory

Sorry to bother you again. Actually, when I was using SyncSN, I got some different errors. I tried to imitate the way you used in train.py, but my model outputs...

@ekelsen , hello, I'm using Keras(backend: tensorflow 1.12) and cuda9.0, but the train speed is still slow with this loss function. Can you give me advice? My GPU is GTX...