Training starts but no updates are dumping to checkpoints?

[JordanDavis-1]

Not sure if this is a wavegan related problem or my own install of tensorflow / python. The training will start but only dumps "model.ckpt-0" etc.. and never updates.

Any ideas? Thanks!

[markhanslip]

Hey Jordan,

I've just revisited this codebase and I'm having a similar problem. It was working fine for me middle of last year but now it progresses incredibly slowly - checkpoints are being dumped but at a dramatically lower rate than previously. What's confusing is that the model is clearly inhabiting the GPU but is not being trained at GPU-like speeds. The same behaviour is happening locally and in Colab.

I'll update if I manage to figure out a workaround or cause.

Cheers