GDL_code icon indicating copy to clipboard operation
GDL_code copied to clipboard

Published 20 hours ago • verified publisher icon davidADSP

Question: 05_01_cyclegan_train extremely slow (TF 2.0)

Open fnpdaml opened this issue 5 years ago • 3 comments

Hi there,

Training the 05_01_cyclegan notebook (apple2orange) is taking me ~30 hours on a system with a 2060RTX Super. Is anyone else having similar (large) training times?

I tried Google Colab - it runs, but it takes so long it just times out.

What are your experiences and thoughts?

Thanks!

fnpdaml avatar Jan 15 '20 12:01 fnpdaml

I just tried with my GTX1060 using TF 1.10

Took around 15 min to finish 2. epoch and then I stopped it.

So it takes 4-5 min to finish a epoch in my case...

ygtmesci avatar Mar 20 '20 00:03 ygtmesci

200 epochs of apple2orange using TF 2.0 takes about 7 hours on a Google Colab kernel with a GPU accelerator so it was either a T4 or P1000 Nvidia GPU. This works out to about 2 minutes per epoch, so it sounds like it varies by device.

DCtheTall avatar Apr 14 '20 12:04 DCtheTall

apple2orange notebook - I use 1x GTX 1080 Ti - Core™ i7-9700K - 13.7 TFlops. - TF2 one epoch 3 min 33 seconds.

TVI-BIZ avatar Jun 16 '20 10:06 TVI-BIZ