fedora-coreos-pipeline
fedora-coreos-pipeline copied to clipboard
coreos
Send email and/or IRC notification if pipeline starts failing
E.g. to freenode/#fedora-coreos. Otherwise, it's too easy to miss when status goes red.
Would it be fine to use a plugin for this? The plugin https://github.com/jenkinsci/email-ext-plugin seems to be referred to a lot when searching on the topic. Would just need to make sure the version of the plugin works with the version of the Jenkins console we are using.
Also looks to be an IRC plugin https://github.com/jenkinsci/ircbot-plugin, but not as recently updated.
Yup, using a plugin is definitely the way to go! I have a slight preference for IRC over email for this, but if there's blockers there for whatever reason, email is fine to start.
Re. versioning, that shouldn't be an issue. The master is more or less up to date with the latest LTS release. :)
:+1: I like the sound of IRC as well, avoids the extra noise to a mailing list (though wouldn't expect it to be too noisy itself).
For a channel to log to, would we eventually want a separate channel for the Jenkins pipeline rather than #fedora-coreos, e.g. #fedora-coreos-jenkins?
I could give this a try if we'd like it soon - could try getting a POC working locally with the Jenkins IRC plugin.
For a channel to log to, would we eventually want a separate channel for the Jenkins pipeline rather than
#fedora-coreos, e.g.#fedora-coreos-jenkins?
Depends on how chatty it would be. We could start with #fedora-coreos and then move to something like #fedora-coreos-releng if it was too chatty.
Did some looking into this last week - I could install the IRC plugin [1] using the Jenkins web console. However, there isn't a tab in the web console for configuring "Post-build Actions"> "IRC Notification", when looking at the project configuration (described in [5]):
The configuration I have done (in my local developer pipeline) is:
Right now, the IRC bot joins the specified channel when Jenkins starts, but doesn't send notifications when a build starts.
Looking for a way to configure the IRC plugin through the pipeline, it seems support for this is not available [2]. An alternative which may be worth considering is [3], sending the IRC notification through the Groovy script. This may be enough for what we need - and it'd be nice to define everything in the Jenkinsfile. An example implementation is available [4], if we can make it work with freenode.
[1] https://wiki.jenkins.io/display/JENKINS/IRC+Plugin [2] https://issues.jenkins-ci.org/browse/JENKINS-33922 [3] http://davehunt.co.uk/2016/11/25/irc-notifications-in-jenkins-pipelines.html [4] https://github.com/jenkinsci/pipeline-examples/blob/master/pipeline-examples/ircnotify-commandline/ircNotify.groovy [5] https://plugins.jenkins.io/ircbot
This also looks promising: https://github.com/jenkinsci/ircbot-plugin/pull/21
This also looks promising: jenkinsci/ircbot-plugin#21
Hmm yeah, though as per the discussions there and https://plugins.jenkins.io/ircbot itself, it doesn't seem like the plugin is maintained much anymore.
[3] http://davehunt.co.uk/2016/11/25/irc-notifications-in-jenkins-pipelines.html
Oh man, that one made me giggle. I guess that'd work, though not sure if we want to resort to that.
Hmm maybe for now, let's at least do Slack notifications? It has native pipeline support, and is actively maintained. Even once we add IRC support, I think having it on both could be useful?
For IRC, once we get fedora messaging working, I wonder if there's a bot in the Fedora infra which could do what we want already (i.e. fedmsg to IRC). We should ask the infra folks about this.
In RHCOS land we've discussed using the Slack plugin as well, but decided we'd rather not get involved with Jenkins plugins if possible. We'll probably implement something like this instead:
https://www.geekytidbits.com/til-post-to-slack-from-jenkins/
For IRC, once we get fedora messaging working, I wonder if there's a bot in the Fedora infra which could do what we want already (i.e. fedmsg to IRC). We should ask the infra folks about this.
yes. notifications from fedora messaging /fedmsg go to IRC channels all the time. You can also configure them to be sent to you directly. See https://apps.fedoraproject.org/notifications
Hi all, just FYI : I recently joined the list of maintainers for the Jenkins instant-messaging-plugin (the core doing grunt work) and the ircbot-plugin (the protocol adapter for IRC in particular), did some cleanup and merges of outstanding PRs, and had cut releases of both.
So it should now be possible to use an ircNotify() step from your pipelines, if you'd still go that way (and some caveats apply compared to legacy freestyle post-build step setup, at least for the current state of the plugin codebase, as detailed in the README and Jenkins Wiki https://wiki.jenkins.io/display/JENKINS/Instant+Messaging+Plugin and https://wiki.jenkins.io/display/JENKINS/IRC+Plugin pages). Note that if your Jenkins master essentially serves one project and team/community, you can define the channels to connect and notification strategy in the global configuration, and then the ircNotify() step needs no parameters (but needs empty parenthesis instead, per groovy spec). For special cases you can customize any of the half-a-dozen toggles there are.
On a side note, something perhaps lacking conceptually with Jenkins notifications about job results, is that you'd have to add notification steps (whether calling existing plugins, or making your own) separately for each protocol (IRC, jabber, mail, etc.), in each job definition. There is no magic "notify-everything-configured" silver bullet currently (a shared library defining the magic notifier for your build farm, so you only call one routine and maintain it in one place, might come close however).
There now exists a matrix plugin for jenkins: https://github.com/jenkinsci/matrix-communication-plugin
since we are moving more and more away from IRC because of the libera/matrix bridge being down maybe we should explore this as an option.
Yeah, if the outcome of https://github.com/coreos/fedora-coreos-tracker/issues/1566 is to fully cutover communications to Matrix, then I think targeting Matrix here too seems like the most reasonable path now. It also supports features that we've come accustomed to having for these notifications (like threading).