[ENH] sleep compositor for testing and simulation

[fkiraly]

The test surfaces in surfaces have an argument sleep that introduces latency, e.g., for testing purposes. https://github.com/SimonBlanke/Surfaces

I think this is a neat idea, in hyperactive I would deal with this as a compositor: a class Sleep, which can be instantiated

Sleep(optimizer, sleep=42), this creates a composite that adds a delay to each run of solve.

Thoughts on the design, @SimonBlanke? (imo low priority for now but just capturing my thoughts)

[SimonBlanke]

The idea behind the sleep parameter is indeed for testing/benchmarking. Some mathematical test-functions have interesting shapes, but are evaluated very quickly. Adding a sleep parameter enables to imitate the computational load of machine-learning training or engineering simulation. For me it is more intuitive to imitate a computational expensive objective-function, then slowing down each optimization iteration inside the solve-method. We also don't need both, because optimizers can already me compared without taking into account the optimization time by just comparing the number of iterations of the algorithms. For example: What is the optimum score reached by hill-climbing compared to bayesian-optimization after 100 iterations. The fact, that bayesian-optimization takes much more time to select the next position to evaluate is ignored in this example (with might be desirable).

The API could be realized by adding a simple sleep parameter to the base-experiment (init?).

[fkiraly]

What would be the use cases for the sleep feature? If testing only, I do not think it should be in the user facing API.

[SimonBlanke]

I described a use case in the comment above:

Some mathematical test-functions have interesting shapes, but are evaluated very quickly. Adding a sleep parameter enables to imitate the computational load of machine-learning training or engineering simulation.

Since the sleep "unnecessarily" slows down the evaluation, which should not be desirable under any normal circumstances. "Normal" meaning achieving an optimization goal as fast as possible.

So my conclusion is, that the sleep parameter is only useful for testing/simulation.

[fkiraly]

So my conclusion is, that the sleep parameter is only useful for testing/simulation.

Ok, then I have the very strong opinion that it should not become part of the core API function signature.

Of the solutions that do not do that, the compositor as proposed is one solution. Another is adding callbacks, see #158, which allows the user to inject sleep commands.