github-bot icon indicating copy to clipboard operation
github-bot copied to clipboard

Published 20 hours ago • verified publisher icon nodejs

Bot comment on build failure

Open mmarchini opened this issue 5 years ago • 5 comments

(not when the job fails to start, but when the tests or build fails, so it wouldn't revert https://github.com/nodejs/github-bot/issues/238)

node-core-utils has this great tool ncu-ci pr, which can be used to get a list of failing tests. We could use the GitHub Bot to comment the result of ncu-ci pr back on the Pull Request when a build fails, so it's easier for folks to see which errors happened. Or the bot could edit it's own comment, to avoid notifications, but personally I would like to get notified (I understand folks might be opposed to it though).

mmarchini avatar Mar 11 '20 00:03 mmarchini

To ensure I understand you correctly and having in mind I haven't used ncu-ci before; in addition to the inline PR statuses shown below, the Github Bot would also post a comment providing more details about the actual test(s) that failed? Meaning not just the fact that a job failed like the PR statuses says?

Screenshot 2020-03-11 at 09 14 05

phillipj avatar Mar 11 '20 08:03 phillipj

Exactly. It would look like this: https://github.com/nodejs/node/pull/32116#issuecomment-597683266

Edit: I updated all Bot comments on https://github.com/nodejs/node/pull/32116 to also include the failures, if anyone wants to see how "the bot could edit it's own comment" would look like.

mmarchini avatar Mar 11 '20 14:03 mmarchini

I just thought of two other huge benefits of this:

  1. We'll keep a history of CI failures (since Jenkins deletes old CI runs)
  2. When searching for a flaky test, every PR where the test failed will show on the results (which will make it easier to see if the test is flaky than navigating Jenkins to find each occurrence of that test failing).

mmarchini avatar Mar 11 '20 17:03 mmarchini

Well said!

I definitively see the value of trying what you're proposing. Learning from well intentioned changes we've introduced before, that ended up getting reverted later primarily because we didn't reach out to core collaborators beforehand, I'm wondering if we could do some kind of collaborator vote or similar before jumping onto implementing this?

phillipj avatar Mar 15 '20 19:03 phillipj

That might be a good idea. I also think the "edit comment" idea would be more likely to be well received. I'll start a poll, thank you for the suggestion.

mmarchini avatar Mar 15 '20 19:03 mmarchini