mlos_bench: Trial repetition

[yshady]

trafficstars

I would like to run each config 30 times but not 30 at same time. For example, a default config should be running 30 times but not concurrently.

Currently trial repeat param for mlos_bench will run the trial repeat number all at the same ts_start

[bpkroth]

Chatted about this, there is some confusion over some storage entries, but they aren't running concurrently (yet).

We should

• [ ] add ts_create to the DB to note when a trial was "created" vs when its (scheduled to) start(ed)
• [ ] add support for specifying "(random) splay between" trial repeats in the configs
• [ ] enable the scheduler to only start trials with current or past ts_start

[bpkroth]

@motus I don't think all of this is handled yet, so reopening.

[yshady]

@eujing can you see if trial repeat with master is still broken? I observed it acts like a minimum bound and not as a maximum

[eujing]

@yshady trial repeat seems to be working at an exact value when using the main branch. I have been using 1, and getting new configs proposed every time. Maybe this is not an issue anymore in the main branch.

[yshady]

I was using 5 or more

[bpkroth]

Should probably just write a test for this.