Frank

Hi, we will gladly look into this! To me, it is a bit unclear what you mean by you "have no explicit loss function class"? Could you perhaps provide a...

I am closing this issue as it is stale. Please feel free to open it again if you have further questions.

I am closing this issue as it is stale. Please feel free to open it again if you have further questions.

Hi Cristian, to reproduce our results, please take a look at https://github.com/fsschneider/cockpit-experiments This repository should hold all the information and code to replicate our experiments. Let me use experiment `07_learning_rate_selection`...

I am closing this issue as it is stale. Please feel free to open it again if you have further questions.

Hi, you could take a look at our [experiment repository](https://github.com/fsschneider/cockpit-experiments), which contains all the code to re-create the experiments shown in the paper, including the animated showcase. More specifically, the...

Hi @pratt3000, the DeepOBS package is no longer maintained. I highly recommend using the [**AlgoPerf** benchmark suite](https://github.com/mlcommons/algorithmic-efficiency) instead (I helped develop it). In my opinion, it is better in pretty...

Hi Niccolo, we believe that figuring out _before_ the call of `update_params` whether an eval is coming up is a bit dangerous. E.g. a submission could then in the last...

We plan to discuss feature requests like these in the benchmark code during the WG meeting on Thursday, 9/5.

While thinking about this issue and the possible changes it required, I created this skeleton version of our current `submission_runner`'s logic. This is a simplified version of the **current version**:...