Feature: TrialMetricsPlottingObserver should support various plot functions for various metric type

[vincent-antaki]

Is your feature request related to a problem? Please describe. TrialMetricsPlottingObserver is a neat subscriber-listener interface designed to be used with a BaseHyperparamsRepository instance. However, it assumes that every metric saved by the hyperparams repo is either a float or an int and that, for each metric, it should plot one line for the train and one for the validation (e.g. a loss/error, an accuracy or a score).

However, metrics could potentially be more complex data structures (e.g. a confusion matrix). To deal with this, I've temporarily added a few lines in #435 to ignore every metric that isnt a float or an int. Eventually, we should re-write TrialMetricsPlottingObserver to be a bit more flexible.

Describe the solution you'd like I'd replace all arguments of the constructor except the folder_name by a "plotting_function_map" which would be the following mapping : {"metric_name":str -> callable_on_complete:Optional[Callable,None], callable_on_next:Optional[Callable,None], render_action:bool = False}

It would probably be a good idea to add some default elements to the map for some already implemented MetricCallback.

Describe alternatives you've considered Alternatively, we could have an object like TrialMetricsPlottingObserver for every type of metric we wish to plot. With the right abstractions, it wouldnt be that much more lines of code than the current solution I suggest. However, I think it would be less accessible / an out-of-the-box solution.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs in the next 180 days. Thank you for your contributions.