Adding ability to add attachments to testcase in junit plugin

[Zechtitus]

Context

I was not able to put attachments on a testcase with the junit plugin. This pull request will look for any file attachment that matches the classname.name of the testcase

i.e. if you have a classname of 'com.someTest' and name of 'shouldAdd' you will be able to add attachments to the test case.

• com.someTest.shouldAdd.txt
• com.someTest.shouldAdd-somepic-1.png
• com.someTest.shouldAdd-somepic-2.jpg

Unit tests provided

[CLAassistant]

All committers have signed the CLA.

[noomorph]

@A1exKH , @baev , I might take this if you are really interested in accepting this PR. Shall I?

[A1exKH]

@A1exKH , @baev , I might take this if you are really interested in accepting this PR. Shall I?

@noomorph, in my opinion, this PR adds good functionality, it is really useful.

[baev]

Is there any framework that uses the suggested convention to provide additional information for JUnit reports? If not, I suggest not introducing a new undocumented feature to an undocumented format (which already has many dialects).

Regarding the implementation: Allure now aims to be able to process test data in real time (done in Allure TestOps, planned for Allure 3).

To achieve that in the most efficient way, we need to:

1. Make readers stateless
2. Process results as blobs (byte stream) with metadata, not a file system object
3. Do not rely on other files that may or may not be present in the real-time system.

The current feature relies on file name convention to consider files to be attachments, so the only way to implement that (considering the points from above) is to each time after a new test result arrives, look through all already received result files and find all possible attachments, and each time when new attachment (basically, file that not accepted by any reader) arrives do the same, looking through all the results to find the one with the desired method name.

Then, looking through the suggested attachment file naming convention: ${classname}.${methodname}-${attachmentname}.${ext}

Since junit.xml is widely used in different languages and test frameworks, there is no guarantee that classname, methodname, or attachmentname will not contain special characters.

e.g.

my.first.test.and-more.or-not.tar.gz could be one of

class name	method name	attachment name	ext
my.first.test	and	more.or-not.tar	gz
my.first	test.and-more.or	not	tar.gz
my.first.test.and-more	or	not.tar	gz

and so on...

So, we would not accept this PR in the current state.

[noomorph]

@baev , thanks for the elaboration, it was an interesting reading 😊

[Zechtitus]

with Allure 3 -- would the above flow for adding documentation be available? It's been a long while but I could look at extending the PR to accomodate possible changes. being able to add extra documentation to the test flow has been very useful over the years.

The usage of '.' could possibly be exchanged with a '%%%' delimiter or some other format to separate out the various fields,

based on this my.first.test.and-more.or-not.tar.gz

maybe the best approach is just to prefix each section like so (className)my.first.test(methodName)and(attachmentName)more.or-not.tar.gz

[baev]

I think the best way to implement it is to introduce a reserved property name that will store the link to an attachment, something like:

<?xml version="1.0" encoding="UTF-8" ?>
<testsuite tests="1" failures="0" name="test.SampleTest" time="1.051" errors="0" skipped="0">
  <testcase classname="test.AttachmentsTest" name="shouldAddAttachment" time="1.051">
    <properties>
      <property name="_attachment" value="expected-attachment-file-name.xyz" />
    </properties>
  </testcase>
</testsuite>

However, why wouldn't you simply produce test results in Allure format instead? It's super simple and supports attachments out of the box.

[Zechtitus]

I seem to remember looking at this but wasn't able to get it working with jenkins. The plugins wouldn't pick up the attachments.

That was however 4 years back so possible it's working now?

actually - the problem I think was that we were dealing with junit.xml output - and that is restriced in what fields you can have. Looking at this - would we be able to have both junit output as well as allure.json for test cases together?