pytorch-pcen icon indicating copy to clipboard operation
pytorch-pcen copied to clipboard
verified publisher icon daemon

nan during training

Open cyrusasfa opened this issue 4 years ago • 1 comments

thanks for sharing this.

I am trying to train PCEN in a torch layer. However, at unpredictable points during training, the loss collapses to nan when this is introduced. I'm thinking that this must be due to certain parameter configuration, but I cannot trace the bug.

Thanks

cyrusasfa avatar Jan 05 '21 20:01 cyrusasfa

Is this with a CTC objective? Have you isolated the issue to PCEN? Is the PCEN frontend trainable, or is it fixed?

daemon avatar Mar 02 '21 18:03 daemon