Allow automated sending of mirror issue notification

[jelly]

trafficstars

Talking about this on IRC lead to the conclusion that it would be easier to just integrate the mail sending directly into archweb. Thus the changes above are not needed, but there needs to be some service that monitors the mirror check data and sends notifications like this:

• notify about "out of sync" if the lastsync time is older than 72hours
• notify about "unreachable" if the lastsync file could not be fetched for 24hours
• notify if the lastupdate file is 72 hours older than the lastupdate file on our master mirror

A potential improvement is to be able to set the timeouts for each mirror on their own (some mirrors sync every minute so 1 hour delay might be more appropriate than 72).

If a tier 1 mirror is unreachable or out of sync, tier 2 mirrors that use this mirror should NOT receive emails about being out of sync. There is a field in the database that records the source mirror for tier 2 mirrors, but this field is often not correct due to mirrors changing their upstream without notifying us.

Templates for the email texts can be found here: https://git.server-speed.net/users/flo/arch-mirror-tools/tree/bin/generate-mirror-mail.pl

Original ticket https://bugs.archlinux.org/task/45556

[jelly]

A new proposal has come up which proposes:

1. Send emails to mirror admins (define them in archweb)
2. Store "flagged" mirrors, make the script check if flagged mirrors are resolved
3. Mail with a link to a page with an example mail

[Bluewind]

Just to clarify further: My idea is that eventually archweb automatically sends notifications to mirror admins when a problem is detected. Obviously this needs some logic to determine when a problem starts/what a problem is. However, since problem detection may be somewhat complicated and initially buggy, the first step towards this is to send the notification to us and then we have some button or link in the mail to send it to the actual mirror admins. When we are confident that it works properly, we can change the recipient of the initial notification and remove the button/link.

Apart from the tier 1/tier 2 issue mentioned in the initial comment, another issue that comes to mind is mirrors that conduct maintenance. Sometimes they notify us prior to the maintenance so right now we put in a note and hold off on sending them mails when we see the mirror being down. I'm not sure how to best handle this in an automated notifier. I guess we could just ignore it for now. Maybe make sure that we don't spam them by only sending a maximum number of notifications (2 or 3?) per day/week.

[jelly]

Another idea is setting the mirror inactive when it has errors like DNS resolving issue and email the mirror admin that his mirror has been disabled and he should re-email to get it activated.

[peanutduck]

Hi I would like to work on this issue, any pointers for me to get started?

[jelly]

I've pushed my WIP commit I had locally https://github.com/archlinux/archweb/commit/adc2cf7a6bc627f4de3243c841da590c7d35ddca

The idea is, that there is an overview of mirror issues, the mirror maintainers (A group on archweb) can press on a "send" or "cancel/remove" button. The send button sends a templated email (see the perl script from Bluewind above) to the mirror administrator.