Tensorboard hparams plugin API reports metrics from final epoch, *not* epoch with best performance on metric tracked via EarlyStopping

[fPkX6F1nGTX]

System information

• TensorFlow version: 2.3.0
• Are you willing to contribute it: Yes, with guidance/help to know where to look

Describe the feature and the current behavior/state. Feature would change current behavior of Tensorboard. Current behavior is that Tensorboard displays the validation loss from the final epoch of training, which is not useful when comparing different models to each other.

Will this change the current API? How? This will change tensorboard.plugins.hparams.api to report the performance of the best training epoch, not just the final epoch. Since the purpose of validation is to detect overfitting, this is in line with the reasons for performing it.

Who will benefit with this feature? Everyone who uses Tensorboard to compare models to each other based on the performance of the best training epoch.

Any Other info. I am currently using a standard tf.keras.callbacks.Tensorboard instance, sub-classed with the following method as the only modification:

    def on_train_end(self, logs=None):
        if os.path.exists(os.path.join(self.log_dir, "train", "plugins")):
            shutil.rmtree(os.path.join(self.log_dir, "train", "plugins"))

I am also creating a callback with the following code:

from tensorboard.plugins.hparams import api as hp
hp.hparams_config(
    hparams=hparams_list,
    metrics=[hp.Metric(CategoricalAccuracy().name, display_name=CategoricalAccuracy().name)],
)
hp_callback = hp.KerasCallback(writer=output_dir, hparams=session_hparams)

[rmothukuru]

@brethvoice, Can you please share the code corresponding to the statement,

Current behavior is that Tensorboard displays the validation loss from the final epoch of training.

so that it helps us to reproduce the issue better and understand exactly what is missing.

Thanks!

[fPkX6F1nGTX]

@brethvoice, Can you please share the code corresponding to the statement,

Current behavior is that TensorBoard displays the validation loss from the final epoch of training.

so that it helps us to reproduce the issue better and understand exactly what is missing.

Thanks!

The code is over 300 lines, so I posted it to my only Github repo. It can be run from Windows 10 using local GPU if available.

Note that on line 253, I have restore_best_weights=1 which means I am reporting back to Optuna based on the model trained with lowest validation loss. All epochs after that point are ignored, and that validation loss is what I want to view in TensorBoard.

[psybuzz]

Thanks for this feature request. It seems reasonable to want to see the "best" (maximum or minimum) value of a particular metric in the table UI at a glance, rather than just the value from the final epoch/step.

If I understand how the Hparams dashboard works, the metric names are linked with series line charts shown in the Scalar dashboard. If that is the case, and the scalar + hparam data are already plumbed to TensorBoard today, I think this might not need any API related changes. As such, this might only require frontend fixes.

[fPkX6F1nGTX]

@psybuzz if we are using another metric besides validation loss to trigger EarlyStopping, it would be preferable to report on the performance of the epoch with the best performance in that metric. That is what is interesting to the researcher.

[fPkX6F1nGTX]

Here is a code snippet which might help as an example of a current workaround:

metric_data = model.fit(train_dataset, validation_data=valid_dataset, callbacks=callbacks, **trainer_settings)
if tuner_settings["direction"] == "maximize":
    best_metric = np.max(metric_data.history[tuner_settings["metric_tracked"]])
if tuner_settings["direction"] == "minimize":
    best_metric = np.min(metric_data.history[tuner_settings["metric_tracked"]])
best_metric_epoch = [e for e, metric in enumerate(metric_data.history[self.hp_tuner_settings["metric_tracked"]]) if metric >= best_metric]
for metric in metric_data.history.keys():
    trials_info[trial_name][metric] = metric_data.history[metric][best_metric_epoch[0]]

We use the trials_info dictionary later on, but it could become part of the front-end or API of Tensorboard.

[nfelt]

Thanks for the details @brethvoice. I think it might make the most sense as a UI option that lets the user dynamically select whether to show the latest, max, or min value of the metric as the representative value. Here's a mockup:

Does that understanding match your use case?

If you're interesting in contributing this feature I can provide some pointers on the relevant code that would need to be changed.

[fPkX6F1nGTX]

@nfelt yes that is the use case exactly. I unfortunately do not have time to contribute; does that mean I need to close the issue? I am new to Github and not wanting to commit a faux pas.

[psybuzz]

does that mean I need to close the issue?

Not at all! To chime in on the thread, having a way to view the "best" value of a metric in the Hparams dashboard sounds like a very reasonable request, and keeping this issue open is helpful for the TensorBoard team to keep track of what issues are most important to users.

In general, feature requests are typically only closed when it is irrelevant to TB, a duplicate of another issue, obsolete, already fixed in a newer version, or not a product priority for the TensorBoard team etc. On the other hand, an issue remaining 'open' does not necessarily mean that it will be fixed soon, given that the TensorBoard team has to balance priorities and work with limited resources.

Contributions are welcome, but are certainly not expected as an obligation to users.

[claverru]

it's been 2+ years since last comment, still this feature not implemented

[bmd3k]

Indeed we haven't been able to prioritize this. It still seems like a valuable feature to build. We do welcome contributions from the community if you are interested in building it yourself. If not, no problem.

[claverru]

@bmd3k I think it's going to be pretty hard to find people that need this feature and are also capable of developing it :)

I've just taken a look and saw that it's built in Angular. I can try but never did any serious front development before.

[bmd3k]

That part of the web application is actually built in Polymer (not Angular), which is probably still foreign to you. And the backend/http parts are built in Python, which I suspect is much more familiar to TensorBoard users.

https://github.com/tensorflow/tensorboard/tree/master/tensorboard/plugins/hparams

But, generally, your point makes sense to me.