Training got struck

[indhu26]

Hi, I tried training with the train.py for both custom datatset and VOC dataset. But what I observed was that it got struck in keras fit call.

For VOC dataset - Epoch 1 got completed , but 2nd epoch got struck. No logs were printed nor progress was seen. Left for more than 6 hrs, no improvement

For custom dataset - Same was observed in epoch 1 itself.

Can you tell if there is something going wrong on my side ?

Thanks in Advance

[liborvaneksw]

Hi, I experienced some problems with dataset optimizations in tf 2.4 recently. The training did not even start. Could you try to turn them of? Please let me know if it helped. (I did not investigate more yet so I don't know which part caused the problem.)

options = tf.data.Options()
options.experimental_optimization.apply_default_optimizations = False
ds = ds.with_options(options) # ds: tf.data.Dataset

Libor