Clarify hydra-notify `eval_failed`

[Rowan-Mather]

Is your feature request related to a problem? Please describe. From the manual entry on hydra-notify, my understanding was that an event for eval_failed is triggered when you get an evaluation error such as the one below. But this does not seem to be the case.

Perhaps this is only triggered when hydra cannot perform an evaluation of any description - irrelevant of whether it results in an evaluation error or not?

Example evaluation error from the #tabs-errors page:

evaluation failed with exit code 255

Describe the solution you'd like Please be specific in the manual.

[Rowan-Mather]

From further experimentation, this is my new understanding:

• eval_failed is triggered when no evaluation can occur at all upon the 'check' of the jobset.
• lastChecked is the last time the evaluator checked this jobset, and should be triggered on input update, and then periodically.
• Sometimes you might have a jobset which has some jobs that can be evaluated and some that cause evaluation errors - these would not trigger eval_failed.

So there doesn't seem to be any obvious way to determine whether a jobset has evaluation errors, save checking both eval_failed and every single completed evaluation for errors. NB that if you hook into eval_failed you will get repeat reports of the same errors until they're fixed, as hydra periodically checks the jobset.

You have two routes to do this:

1. Integrate with hydra-notify, which is what is recommended in the manual, but this requires patching hydra with your desired functionality here: https://github.com/NixOS/hydra/blob/master/src/lib/Hydra/Plugin.pm
2. Use the DB directly. Not recommended but doesn't require patching hydra and is simpler in my opinion.