Show max instead of last value in runs

[Pyrestone]

🚀 Feature

In the runs explorer, I would like an option to show the max/min achieved value of a metric in the run list instead of only seeing the last value.

Ideally I would also like to sort by those.

Motivation

I would like to know the best validation set accuracy of all my models without having to check each time series individually. Ideally, I would also like to be able to sort by this.

Pitch

click the 3 dots at the top of a column on the runs -> submenu opens, where I would like to have options "show min", "show max", "show first", "show last", and ideally an option "sort by this column"

Alternatives

selection via "manage columns" would also be possible, so I could see last and max simultaneously.

Additional context

Add options

• "show min"
• "show max"
• "show first"
• "show last"
• "sort by column" (switch ascending/descending)

to this menu:

[gorarakelyan]

hey @Pyrestone, thanks for the feedback! Will prioritize this enhancement for the next milestone(v3.13).

[Pyrestone]

@gorarakelyan Is this still on your roadmap? I would still find this feature very useful

[gorarakelyan]

@Pyrestone sorry for a late reply.

It has been tough weeks in terms of number of critical issues and enhancements that we tried to build and ship for the community. Sometimes we are forced to prioritize even between critical issues.

Currently we are really packed with issues for this and for the next week. The next week particularly is occupied, as we are planning to release Aim v3.14.0 (with revamped home page!), which requires both pre-release and post-release procedures.

Thus, I can not tell for sure if it will be possible to take over this issue within the next 10 days (tough realization 😞). But, I totally agree on the importance and value of this feature request. It is for sure on the roadmap. It is only the matter of time and capacity.

However, we would do our best to build and ship it as a patch release of 3.14 version.

Hope this makes sense to you. 🙏

Thanks for your help and contribution to the community!

[Pyrestone]

Hi @gorarakelyan, I understand that it's more important to focus on usability breaking issues first, (such as #2188 i presume?) It's more important to me as well to have a stable, working, and portable version of aim than having fancy features.

I'm thankful for all the great work you and the team are doing on aim :pray:, keep going!

[gorarakelyan]

Thanks for the kind words and understanding @Pyrestone.

wrt #2188 yes: it is already in progress. The fix is planned to be shipped with Aim v3.14.0 at the end of this week. 🎉

[Castavo]

Hello I'd like to weigh in to say this feature would be very appreciated ! It would also make the "Params Explorer" a more powerful tool !

[inboxedshoe]

Is there any update on this btw? I really appreciate the work you've already put in as this library is now my standard. Would definitely make it easier to be able to view the top performances directly.

Also one option for people still waiting is to log the top/lowest scores separately as a different metric. That would at least help keep track and would give you the sorting option as well.

[Pyrestone]

It's been more than a year. Any updates on this feature?

[sharathmk9]

This feature will be great addition to Runs Explorer. Thanks

[iRuizeWu]

I need this!!

[Pyrestone]

While #3094 adds an approximate implementation of this feature, there are still two important shortcomings of the current implementation:

1. It does not work on runs which were logged with a previous version of AIM (simply shows "--"). I understand that this would require recomputing min/max values on stored data, which may not be desirable. I can live with this, but a workaround to enable this display for older runs would be nice.
2. It's currently implemented as a dropdown for ALL columns, instead of a selection for each individual column. This is very inconvenient because some metrics (e.g. loss) have a meaningful minimum value, whereas others (such as accuracy) have only a meaningful maximum value. Therefore, I would like to be able to select this on a per-column basis.

The second point is most important to me, and it's a simple change in the front-end. It should be possible to implement this relatively easily/quickly (I hope) with the available data from the storage already being in place.

[Pyrestone]

@KaroMourad Any updates on whether finalizing/fixing this feature (implementation on per-column basis and sorting by column value) as described in the initial issue post is still on the roadmap?