Tensorboard callback is blocking process

[chenmoneygithub]

As pointed out by our user, the Tensorboard callback is doing blocking I/O, which means the training is halted until the writing finishes. This creates a performance bottleneck especially when writing to cloud storage.

[mgraczyk]

It's been a while since I've looked at this code, but I think this is what happens.

tf.summary.create_file_writer creates a _ResourceSummaryWriter https://github.com/tensorflow/tensorflow/blob/v2.9.1/tensorflow/python/ops/summary_ops_v2.py#L559

The summary context manager always calls writer.flush() https://github.com/tensorflow/tensorflow/blob/v2.9.1/tensorflow/python/ops/summary_ops_v2.py#L91

writer.flush() calls flush_summary_writer https://github.com/tensorflow/tensorflow/blob/v2.9.1/tensorflow/python/ops/summary_ops_v2.py#L347

Calls flush op https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/kernels/summary_kernels.cc#L114

^I think this is somehow not releasing the GIL but I haven't specifically checked that, just noticed long delays during training even running callback in another thread

Then it calls flush and InternalFlush in cpp, doing the actual flush. https://github.com/tensorflow/tensorflow/blob/v2.9.1/tensorflow/core/summary/summary_file_writer.cc#L69