Adding more information on "failure description" for JUnit report

[Robbie20]

I found this previous discussion to be relevant but wasn't sure if there were any follow ups to that. As I've been playing more with Caraya one of the things I don't fully understand is how to go about naming, such that I don't lose information of the hierarchy.

Here's a simple test VI: And here's the subVI it's calling: This is just trying to represent an example where assertions get nested. The tree displayed by the GUI makes perfect sense to me:

Where I start having an issue is whenever I look at the JUnit XML. For "Boolean Assertion 2" I start losing information of it's call stack. Now in looking at how Jenkins and Azure DevOps interpret the testcase name and classname attributes, this started to make more sense. However, I also think it would be useful to populate the failure message with the call stack, which is also alluded by both as the description inside the failure/error brackets.

I dug a little into the Test Report.JUnit class, especially the 'Test To Report Entry` VI which looks like this:

Right now it seems that in case of a failure we only write FAIL to the description. However, we have the information for the call stack/test hierarchy readily available to pass instead. I don't imagine this would break someone else's application.

I tweaked my VI a little to generate a failure: Here's what the XML report looks like: Here's what it would look like if we wired the call stack to the failure description:

By the way I've found this video very helpful. So I can always customize things my way, but for this particular update it seemed like a waste to duplicate the existing code rather than trying to push for a small change.

So I guess I'm proposing I can try to make this change, just wanted to see if there was a reason not to, since I couldn't think of anything against it myself.

[Robbie20]

Any interest from anybody to provide any comment?

[francois-normandin]

Hi @Robbie20 ,

thanks for your suggestion and forgive me for having missed the first report. I remember putting it aside for later review and forgot about it.

I think these reports can take a lot of forms. I've read many valid points in this post and others about what would make sense or not to some users. I must admit that I don't know that there is any one proposal that has gained more traction than others, at least in these Github pages. In all honesty, I think I'd like to see concrete Report extension classes built and proposed by the community, and if one of them is gaining more traction than others, it could be introduced into a future Caraya update and become the de-facto standard supported by the framework.

The risk of modifying the base code with any proposal, without a significant amount of feedback into the process, is that there would power users who could get really annoyed by the change (requiring modification of their CI pipeline or deployed applications if they upgraded Caraya). We have a very small community of users (LabVIEW in general) and an even smaller one that actually does Unit Tests... On top of that, a small fraction monitors these conversations on open source projects. All I'm saying is that I wouldn't want to push changes that the power users will want to roll back in a couple weeks.

My advice is to create a project which holds one of more Report class extensions, and if possible, package them as VIP files for the community to have access to those reports easily. I'd be happy to link and advertise any such Community-driven Caraya Report extensions in the Caraya pages and wiki. It does not even need to be back-saved to LV2013... really, any extension would be a great addition to the community's toolkits.

[francois-normandin]

Starting with Caraya 1.4.0, assertion and test properties will be supported. Developers will be able to add verbose explanations, requirementID, hyperlink, measurements or custom key-value pairs to any individual test or assertion.

Example of JUnit result file produced with those properties:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites "framework-name"="Caraya" "framework-version"="1.3.0.137">
    <testsuite name="Automated Test Engine" errors="0" skipped="0" tests="3" failures="1" timestamp="2023-06-04T17:06:24.587-08:00">
        <properties>
            <hyperlink>https://github.com/JKISoftware/Caraya</hyperlink>
            <requirement-id>ID-0001-0001</requirement-id>
        </properties>
        <testcase classname="Automated Test Engine" name="Verbose success"/>
        <testcase classname="Automated Test Engine" name="TestCase1">
            <properties>
                <description>Minimum Rail Voltage</description>
                <eval>voltage >= 3,200000</eval>
            </properties>
            <measurements>
                <voltage>3,3046</voltage>
            </measurements>
        </testcase>
        <testcase classname="Automated Test Engine" name="Verbose failure">
            <failure message="{I can overwrite a Verbose Failure Explanation}">"FAIL"</failure>
        </testcase>
    </testsuite>
</testsuites>

[francois-normandin]

Newly featured in 1.4.0 RC1 https://github.com/JKISoftware/Caraya/discussions/172

Comments appreciated about the Property feature. The example above can now be tested from the release candidate package.

(Not yet released on VIPM Community.)