jenkinsci
Github Checks annotation
Dependencies
checks-api
Feature Request
The checks-api plugin which this junit plugin uses supports annotating files with line numbers and other goodies. Would it be possible to add this. I couldn't find any issues or prs where this might have been explored before which is kinda surprising. Can anyone think of any reason junit should not be annotating checks?
http://www.jenkins.io/doc/pipeline/steps/checks-api/#publishchecks-publish-customized-checks-to-scm-platforms
The junit plugin would need to parse error messages and stack traces for filenames and line numbers. How to do this is not always obvious. In the case of Java,
at x.y.z.C$Inner.someMethod(C.java:123)
means you need to look for a file in the repository x/y/z/C.java typically in some subdirectory like module/src/main/java/ and annotate line 123 (which may or may not even have been edited in the annotated PR).
thx for checking this out @jglick
my junit.xmls have
Maybe. The file format is poorly standardized and each test harness will have its own conventions for how filenames are reported, so the plugin would need to use some heurstics.
The file format is poorly standardized and each test harness will have its own conventions
trust me i know :disappointed:
the plugin would need to use some heurstics
maybe start with something simple like if file exists in the workspace, annotate it? (just throwing stuff out, this could be a terrible idea...)
There’s no guarantee the test file will have been changed in the PR though right? Is there a value in annotating one that hasn’t changed?
Is there a value in annotating one that hasn’t changed?
Generally not, but currently GitHub will display the annotation regardless and you cannot turn this off except by pressing (IIRC) A when viewing the Files tab. (The analogous CloudBees CI integration with Checks explicitly suppresses line annotations which are not present in the PR’s diff hunks.)
the test file
Note that a stack trace (or whatever you have, depending on language & framework) may refer to multiple lines in multiple files, both production sources and tests., any (or none) of which might be the “culprit” of a failure.
Generally not, but currently GitHub will display the annotation regardless and you cannot turn this off
this seems like a dealbreaker, and not worth the confusion for something atm that is just eye-candy.
but what if we gave the end user an option to annotate at their own risk via a junit step param?
something atm that is just eye-candy
Well, this could be really useful. If in a PR you write a test which fails in CI, or if some code you edit in a PR throws an exception during some existing tests, having that be called out right in the Files tab at the point of the mistake would be a lot nicer than having to go through a list of test failures in the Checks tab and manually correlate stack traces to lines of code in your IDE.
very def, this is what i was thinking. sry didnt mean to make it sound completely useless