keras-tqdm icon indicating copy to clipboard operation
keras-tqdm copied to clipboard

on_batch_end incredibly slow

Open Tom-Evers opened this issue 6 years ago • 1 comments

When I run an epoch with about 2 million samples and a batch size of 1, Keras/Tensorflow finishes in about 45 minutes.

When I run that same epoch, with TQDMCallback, it takes over 13 hours.

It seems like there's logging going on during batches by default, which shouldn't be default behaviour. Not sure if that entirely explains it, though.

Tom-Evers avatar Dec 19 '18 09:12 Tom-Evers

Same issue here. The problem seems to be the call to np.mean(logs[metric], axis=None) inside the format_metrics function. When you're training 100 batches per second that log gets big fast.

benp84 avatar Apr 13 '19 17:04 benp84