Try to detect dynamics errors post-segment

[salotz]

In some cases dynamics segments (with OpenMM) can return NaNs which then propagate throughout the other code until something hits it that can't handle NaNs which may be far down the chain from the dynamics itself.

This happens when you've interpreted the units for an input file incorrectly (angstroms in the file, assumed to be nanometers by OpenMM), or when a simulation "explodes" when you've constructed it in an "incorrect" but structurally valid way.

In any case it would be very useful to have some simple checks for NaNs at the end of each segment.

Its unclear what the action should be though. The obvious ones are:

• log a warning
• raise an error

The logging one is nice in case you have specifically expected this (not sure this is a real use case) or is otherwise not actually an issue. If you do get some blowup later you can check the log and see if NaNs were observed, and where/when.

The error is nice in that you know exactly where things went "wrong", however that makes it inflexible to get around if you really need to.

Its also unclear where this check should be done. Should the runner check? Should there be an optional filter post MD (or perhaps a post_cycle hook with custom logic or common options), etc.

At minimum we could start with the runner logging this kind of observation since that is non-invasive and would aid in debugging.