Scalar data loads incrementally each time tensorboard page is refreshed

[DavidJanz]

I'm using tensorboardX 1.14 with tensorboard 1.12 (and tested on 1.10 too with same issue), on Python 3.6.

I find that each time I open up the tensorboard page (e.g. localhost:6006), it shows incrementally more scalar data of a run. That is, if I have 10 events, it might show plots of: --run 1, up to 50k steps. I refresh the page, or just open it again, and then it shows --run 1, up to 100k steps Etc. No errors appear in the tensorboard output in the terminal. Refreshing the page 100+ times will eventually result in the page showing all the steps for all the events (for example, 10M steps for each).

This does not occur if the runs are generated whilst tensorboard is already running and a browser is open -- then they are showing as expected, refreshing automatically every 30s.

Is there anything in the mechanics of how tensorboard loads the data that could explain this? What happens each time I refresh the page that could be causing more data to load? What could happen to make it stop loading the data after say 50k points each time?

[lanpa]

That is an expected behavior of tensorboard. It loads the data on the fly.

As of your second observation, I think the reason is that tensorboard consumes your training procedure's output fast enough so everything is very fresh. (Your training code don't output so many scalars in 30 secs, right?)

[DavidJanz]

Thanks for getting back to me. I seem to have miscommunicated. Consider this:

I have an event file with a lot of scalar data in a folder. I start tensorboard in that folder. No training is running. The data that has already been produced and is just sitting there is being loaded incrementally, instead of all of it being loaded at once.

Does that make the issue clearer?

[lanpa]

Tensorboard reads the event file and fills the read buffer. Once the buffer is full, the data will be pushed to the web service. I believe this implementation is better than reading the whole file and show it at once. For more information, please check the main repository of tensorboard: tensorflow/tensorboard