MSOutput Should not silently tolerate exceptions during rule creation

[klannon]

Impact of the bug MSOutput

Describe the bug A change in Rucio caused tape output rule creation to fail and we missed this for 3 weeks causing 7 PB of tape transfers to pile up.

How to reproduce it Break Rucio

Expected behavior If MSOutput fails to create a rule it, it should trigger an alarm, at least if it fails for multiple cycles.

Additional context and error message None

[amaltaro]

And I just stumbled upon this issue, after resolving another 3-4 weeks outage of rule creation in MSOutput, addressed in this ticket: https://github.com/dmwm/WMCore/issues/12044

I am setting this ticket to Q4 such that we can at least implement an alarm and get notified when the whole MSOutputConsumer cycle is skipped.

[mapellidario]

After a private discussion with Alan, we decided that I can start working on this issue. so far, we agreed on a two-pronged approach:

• [ ] make sure that if MSOutput has trouble processing a single workflow, it continues processing the next workflow and does not break from the loop.
• [ ] Send monitoring data to a new AMQ topic (or re-use and existing topic with a new document type).
  • the idea is to have a single plot with a time series of the number of workflows that MSOutput needs to process
  • if MSOutput fails to process one or more workflows, then it will be visible as a non-zero baseline in this new plot. I like this approached because it worked reliably for CRAB's Publisher monitoring
  • we can setup a grafana alert on this plot (send a message if the number of workflows is above a threshold for longer than X days.)

We considered the idea of having MSOutput sending an alert if it fails to process a workflow N times, but it would require to implement some new logic to keep track of past attempts. Too much effort developing new code when we can achieve the same result exploiting existing monitoring infrastructure.