liangwx

Tsinghua University Beijing, China XiaoliangWang

Results 14 comments of


                                            liangwx

PA_MPJPE calculation failed! svd_cuda: (Batch element 0): The algorithm failed to converge because the input matrix is ill-conditioned or has too many repeated singular values (error code: 55).

看起来其中似乎有一定的必然性的问题在里面，如果只是偶然nan的话，不太应该会在重新加载中间的checkpoint继续训练后很快重新出现nan等问题

PA_MPJPE calculation failed! svd_cuda: (Batch element 0): The algorithm failed to converge because the input matrix is ill-conditioned or has too many repeated singular values (error code: 55).

原来如此。还有个问题就是第一次出现nan之后，应该是loss的nan导致参数梯度的nan，从而使得参数出现异常值，但为什么接下来一些step还有loss不是nan的正常值出现？

PA_MPJPE calculation failed! svd_cuda: (Batch element 0): The algorithm failed to converge because the input matrix is ill-conditioned or has too many repeated singular values (error code: 55).

不是很理解，nan/(nan/1000.)不是还是nan吗？

Do you plan to release the training code?

+1

‹
1
2