Sergei Nikolaev

@JohnnyHan can you please try ``` global_grad_scale: 1 global_grad_scale_adaptive: true ``` also try to remove `clip_gradients`. If it still breaks please attach complete log here. Thank you.

@jazzseow could you upload the commands, their outputs and prototxt files used?

A-ha, seems like a bug: when you run `caffe time` convolution algos get optimized like this: ``` I0719 11:48:43.798629 11106 cudnn_conv_layer.cpp:857] [n0.d0.r0] Conv Algos (F,BD,BF): 'conv3_1' with space 0.08G 63/1...

@vxgu86 seems like a crop, have you tried to alter the values?

@beanliao please attach your prototxt files if possible.

@beanliao thank you. could you please run `nvidia-smi topo -m` and `nvidia-smi topo -p2p n` and paste outputs here?

Hi Timo, good idea, thank you!

Hi @TimoSaemann I looked through it and I can't estimate the time, sorry. I'll update this post when get some news.

Hi @TimoSaemann it's on my plate but there are few urgent bugs to fix before the next release, sorry about this.