Fisher
Fisher
@Amelia0911 你好,我同样用twoPositionPTZCam中的700多张图像做训练,源码基本没改动,但是模型好像没有学习的趋势,changemaps都是一片蓝,loss也没有下降的趋势,请问你对代码有做什么改动吗,label是用gtbinary吗 Edit: 自己改了改模型和超参数,可以训练了
@auvipy what's your recommendation about version?
I think this may be because you have not saved the optimizer parameters
@JiaminRen my code is stuck at dist.broadcast, no error message, backend is nccl. Do you test the train code or what configuration I didn’t do
@JiaminRen I just tried to imitate your train code in face recognition to use SyncSN in my code, but I didn't succeed. I met two problem, frist, rank = int(os.environ['RANK'])...
@JiaminRen I quickly test the face_recognition train.py, unfortunately I met the same problems. I think maybe some system configurations I missed.
@JiaminRen my system is ubuntu18.04 and I use ananconda to install pytorch, program is stuck at dist.broadcast
Thanks, `torch.distributed.launch` can solve problems. But sync way consumes a lot of GPU memory, always out of memory
Sorry to bother you again. Actually, when I was using SyncSN, I got some different errors. I tried to imitate the way you used in train.py, but my model outputs...
@ekelsen , hello, I'm using Keras(backend: tensorflow 1.12) and cuda9.0, but the train speed is still slow with this loss function. Can you give me advice? My GPU is GTX...