Feat/robust retry logic

[mikemhenry]

Description

When running MD, sometimes we get an error that is related to the hardware (GPU) and instead of just throwing an exception, we can try again.

Motivation and context

Parts of the API are gross which is why I've copied and pasted the same bits everywhere, but used a function decorator where I could.

Resolves #928

How has this been tested?

Tested on CI.

Change log

Made simulation running more robust by re-trying the simulation when an OpenMMException is thrown

[ijpulidos]

I think for this one it makes sense to write our own retry decorator and use that in the functions that need it, such as the multisampler equilibrate() or minimize() or similar.We can put it in utils.py and import it where needed in the code.

[mikemhenry]

@ijpulidos I don't think that approach will work (at least until we refactor) since we probably want to control the retry parameters at function run time and not function def time.

Also I need to fix these tests, looks like I've got an inf loop somwhere.

[mikemhenry]

We will take care of this in the refactor