sotabench-api icon indicating copy to clipboard operation
sotabench-api copied to clipboard

Allow to run without cache

Open Randl opened this issue 5 years ago • 1 comments

As discussed in Slack, having option to ignore cache can be useful sometimes.

In particular, if I run model someone already run, the result is returned from cache. However, the runtime isn't fetched (or cached?). Then, it is still shown in leaderboard. In case the original run is in leaderboard it is duplicated, otherwise it appears only once but without runtime. Both seem to me undesirable.

Probably, cache should be attached to a repository, and there should be a way of either clearing it or ignoring it for a run, for example with "Re-run without cache" button.

Randl avatar Apr 27 '20 14:04 Randl

Same problem if build fails half-way, see for example https://sotabench.com/user/EvgeniiZh/repos/Randl/ECANet#latest-results The models which were measured in first build do not have runtime associated with them because it was measured in build which later failed. A simple band-aid would be caching runtime too (that probably would require to invalidate existing caches tho)

Randl avatar May 18 '20 11:05 Randl