Provide link to failing test and line in annotations

[EnricoMi]

Question #32 made me think of linking to the failing test and line form the annotations. Given the commit sha, test file and line, the action could generate a link like this and add to the respective annotation:

https://github.com/EnricoMi/publish-unit-test-result-action/blob/de7f7f0c5f7694846ce69e3384f2e5a03253c141/test/test_publish.py#L35

Such a link gets nicely rendered by GitHub:

https://github.com/EnricoMi/publish-unit-test-result-action/blob/de7f7f0c5f7694846ce69e3384f2e5a03253c141/test/test_publish.py#L35

Providing a few lines before and after gives a nice context:

https://github.com/EnricoMi/publish-unit-test-result-action/blob/de7f7f0c5f7694846ce69e3384f2e5a03253c141/test/test_publish.py#L33-L37

Let's hope that also holds for annotations. It probably won't because annotations do not support markdown.

Resolving the test file path from the test result file will be challenging. This might work best with some configuration like how many directories to remove from the path or which path to prepend.

[EnricoMi]

I see the following issues here:

1. The correct path within the repo needs to be derived from the test result file, which might require some extra configuration
2. This might not work for pull requests from fork repositories as the commit is part of the fork but the check run is part of the target repo. Such a link only works inside a repository.
3. Check Run annotations object (which would be the best place to add this) does not seem to support markdown: https://docs.github.com/en/rest/reference/checks#create-a-check-run

[TWiStErRob]

I've been playing around with attributing JUnit XML to lines in a repo, I got this far: If the top of the stack trace contains a filename, that can be used to identify the line: but of course with real assertEquals this would be inside JUnit or AssertJ, so I agree with additional configuration, where the company package name(s) would be required. I guess where this breaks down is if a non-changed test file fails.

Note: moving from GH check annotations to Sarif uploads may be a way:

[EnricoMi]

I guess where this breaks down is if a non-changed test file fails

The annotations API has exactly that limitation as it only shows the annotation in the Changes tab so only modified failing tests show the annotation. The propose links (see PR description) have the advantage that they reference the commit, not the changes, so they are able to show the content no matter if tests are part of the changes.

[EnricoMi]

Looks like Github can show annotations done to unchanged files (Beta): https://github.com/EnricoMi/publish-unit-test-result-action/blob/master/misc/github-pull-request-changes-annotation.png

[TWiStErRob]

Yeah, I've seen that too! It looks amazing.

What do I need to do to get the annotations for JUnit? I don't remember seeing it for tests yet.

Edit: might be https://github.com/EnricoMi/publish-unit-test-result-action/issues/452

[EnricoMi]

In general, two things are needed:

1. the test result files have to contain the filename of the test case
2. the path has to be relative to the git repo root

[TWiStErRob]

JUnit XML is not famous for those attributes 😞, but I see you support some: https://github.com/EnricoMi/publish-unit-test-result-action/blob/master/python/publish/junit.py#L268-L269, where did you get those names from, because the official schema doesn't have it for Maven? Neither the Ant one. Anyway, it's good to know that if I can "enrich" the XML, it'll "just work".

[dmarkey]

Hey, came across this issue with pytest runs in subdirectories don't have the correct annotation.. thinking of some sed magic to add the prefix so the annotations are set correctly, unless there is an alternative?

[EnricoMi]

@dmarkey I think #482 is what you want. Maybe I have to pick up that piece of work and do it myself.

[EnricoMi]

See #495 for a fix.