Reduce Slack notification noise

[tellyworth]

Alerts to the plugin review channel are very long and noisy. A few simple changes would help:

• Send a shorter summary rather than the full email contents. Just the count of each error type (the sniff name).
• Link to the plugin repo edit page for full details.
• Don't alert on trunk commits for plugins that have a stable tag (only on release commits).
• Add simple unit tests for the scanner output so it's safer to tweak.

We could consider refining it later to put the full scan output in a slack thread, as a reply to the summary alert. But I'm not sure how feasible that is given the Slack API code.

Other ideas? cc @dd32

[tellyworth]

• Only alert about files that were changed in the most recent commit?

or

• Scan both the previous commit and the current one, and alert only on errors that are unique to the new commit.

[dd32]

We could consider refining it later to put the full scan output in a slack thread, as a reply to the summary alert.

This should be possible, but it's not as straight forward given the API being used to send the messages. It would definitely help with the length of the alerts.

One of the problems right now is that the results aren't being stored anywhere, so you don't know what's new, and it has to be re-run next every time you load the admin page (There's some limited caching, but it's not storing, it's caching).

The noise of the channel is problematic, but that's the reason why it hasn't been changed to email the plugin author with the details yet.

Don't alert on trunk commits for plugins that have a stable tag (only on release commits).

While I understand the reasoning there, having it not scan trunk would be the wrong choice I believe, as if someone is using trunk, it'd be good to alert them to issues before a release.

I think perhaps we should store the last scanned details, and only alert on the changes since the last scan - Not based on files, but based on warnings generated, so that future additional rules get run over all files.

[tellyworth]

Yeah storing the last results would enable some refinement in future.

We don't necessarily have to send the same alerts to the channel as to the plugin author. Trunk commits could alert the author only, stable/release commits both the channel and the author.