noisy / aggregation problem: how do we handle the fact that some tuners might evalutae some configs multiple times?

[berndbischl]

this relates to the archive function but also the selection of the final best

[berndbischl]

the problem is that the tuner might run a different resampling on the some config multiple times. now MAYBE we want to aggregate over this? to producde noise-reduction via averaging? but which "grouping" column helps us to do this?

[berndbischl]

@mllg we should still discuss this

[be-marc]

irace shows how to run the same configuration on different resamplings. The tuner adds a grouping variable to the archive and implements it's own .assign_result function.