Overall view on results using better comparison of tables

[VeronikaPolakova]

Proposal Summary

I would like to compare the tables of the experiments (t01, t02, etc.) similarly to the way in the screenshot. It includes:

• merging tables for several experiments (do not compare tables in ClearML as separate tables on the left and right side)
• color formatting (greener experiment is better as it has lower FRR (values) on several FAR levels (rows))

Motivation

Frequently, you are interested in comparing multiple metrics at once. E.g. in binary classification you can be interested in False Reject Rate (FRR) when setting the threshold on various scores. You can report these metrics as scalars, however, you loose overall view on the performance by comparing multiple scalars on separate places. You can also report these metrics as table, but comparing tables in ClearML does not allow you to see experiment results as individual columns of one table. This again makes the comparison of the experiments on multiple metrics more difficult.

Related Discussion

Slack thread: https://clearml.slack.com/archives/CTK20V944/p1661777693843569

[ainoam]

Thanks for the suggestion @VeronikaPolakova.

Note that ,discounting the color code, you can get a similar view if you report these values as variants of the same metric (i.e. 'level_decimal/0.001', 'level_decimal/0.002' etc.) when using the scalar value view when comparing experiments. Does this view provide value for your use case?

Regarding color coding - Identifying the extremes for each row is fairly clear. How were you considering ClearML assigning "better"/"worse" meaning to the values of a user defined metric, or identifying thresholds for transitioning through the color scale?

[VeronikaPolakova]

Hi @ainoam, thank you for your reply. The scalar value view is possible, but IMHO it is still too far for eyes to effectively compare the values in the case of more than two experiments. Imagine having 20 experiments each having multiple FRR values to compare. Using the scalar value view you totally loose the overview as you must using the scroll bar to see them all (see screenshot). Moreover, you are interested in overall performance on multiple threshold positions, so you are forced to do the scrolling multiple times and remember the rank of experiments for each threshold position - we find it really difficult, so we rather use Google Sheets for comparison :disappointed_relieved: What I imagine that user should have the option choosing between the scalar value view and the more compact view I suggest. The color formatting can be probably done in the UI, where the user can choose from multiple colormaps. Considering the color transitions I suppose the edges of colormap represents the lowest/highest value of the row and the values between will be colored based on the distance from the lowest/highest value. Probably, the user should be also able to set the lowest/highest value to overcome color formatting problems in the case of extreme values in the row.

[pollfly]

Hey @VeronikaPolakova! clearml-server 1.13.0 is now out with an improved scalar value comparison.