Add a deterministic Runner strategy

[jbweston]

Currently if you use the default Runner with more than 1 concurrent task, your run will be nondeterministic because the concurrent tasks can finish in different orders.

We could relatively easily add another strategy to the runner where it sends out N tasks initially, then waits for the first of these to come back before issuing a new point, then waits for the second to come back etc.

The advantage of this strategy would be deterministic runs even with >1 tasks running asynchronously, the disadvantage would be potentially inefficient resource usage (imagine what would happen if the first point took ages to come back).

IIRC we wanted to abstract out this concept of "runner strategy" anyway, so this could be a good opportunity to do that.

[jbweston]

Thanks to @tomlaeven for suggesting this

[akhmerov]

This proposal seems sufficiently easy to implement and not interfering with anything, so I'd say we should go for it—nice idea!

We should probably have a parameter queue_points for this strategy, so that the runner asks for n_cores + queue_points to avoid the queue waiting.

Also as a remark, this proposal is not mutually exclusive with implementing deterministic learners and introducing API for learners to run out of suggestions.

[basnijholt]

IIRC we wanted to abstract out this concept of "runner strategy" anyway, so this could be a good opportunity to do that.

We've actually talked about this many times. I agree with this being a good idea.

[jbweston]

Yeah I didn't see a relevant ticket open for this. If I just missed it feel free to link it and close this