NaN error

[JonathanLehner]

I disabled anomaly detection since it crashed. Without it I get the error:

Warning: NaN or Inf found in input tensor. Error: svd_cuda: the updating process of SBDSDC did not converge (error: 3)

this was in epoch 2, it does not always happen. With distort_images: true

[JonathanLehner]

with anomaly detection I get the warning: UserWarning: Anomaly Detection has been enabled. This mode will increase the runtime and should only be enabled for debugging. warnings.warn('Anomaly Detection has been enabled. '

[JonathanLehner]

and after some time I get "RuntimeError: Function 'PowBackward0' returned nan values in its 0th output."

[karfly]

Hi, @JonathanLehner. I also noticed such instability during training an Algebraic Model, but I got such error only after >10 epochs.

[JonathanLehner]

So how can we fix this error? I cannot finish training with it

On Tue, Nov 10, 2020 at 4:05 PM Karim Iskakov [email protected] wrote:

Hi, @JonathanLehner https://github.com/JonathanLehner. I also noticed such instability during training an Algebraic Model, but I got such error only after >10 epochs.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/karfly/learnable-triangulation-pytorch/issues/112#issuecomment-724760606, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFQOPHAVONED6GQPXCSP7GDSPFJEDANCNFSM4ST2FCHA . [ { "@context": "http://schema.org", "@type": "EmailMessage", "potentialAction": { "@type": "ViewAction", "target": " https://github.com/karfly/learnable-triangulation-pytorch/issues/112#issuecomment-724760606", "url": " https://github.com/karfly/learnable-triangulation-pytorch/issues/112#issuecomment-724760606", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { "@type": "Organization", "name": "GitHub", "url": " https://github.com" } } ]