cli Apex Unit Test Reports may generate invalid XML entries

Summary

It seems that there is a corner case that is handled on UI report but not yet for XML Junit reporter. Causing an invalid XML to be generated for some tests with invalid XML texts coming from the message (e.g. System.assert(condition, errorMessage);).

Steps To Reproduce:

Code to reproduce:

@isTest
public with sharing class JsonErrorReportTest {
    @isTest
    static void testExample() {
        String someJsonFromIntegrationOrSomethingElse = '{"key": "value"}';

        Boolean jsonContainsKey = someJsonFromIntegrationOrSomethingElse.contains('"expectedKey"');
        System.assert(
            jsonContainsKey,
            'Error, expected Key : expectedKey, generated JSON: ' + someJsonFromIntegrationOrSomethingElse
        );
    }
}

Execute sync deploy with junit flag (example: sfdx force:source:deploy --metadata="ApexClass:JsonErrorReportTest" --checkonly --testlevel=RunSpecifiedTests --runtests="JsonErrorReportTest" --junit --coverageformatters=text --resultsdir=test-results --wait=5 --verbose)
The generated junit report will have an invalid XML entry.

Note: the same also happens with a regular unit test execution sfdx force:apex:test:run --classnames=JsonErrorReportTest --resultformat=junit --wait=5 --outputdir=test-results.

Expected result

A valid XML such as:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
    <testsuite name="force.apex" timestamp="2022-06-22T17:52:38.040Z" hostname="https://mindful-koala-mjvbtj-dev-ed.my.salesforce.com" tests="1" failures="1"  errors="0"  time="0.05">
        <properties>
            <property name="commandTime" value="0.00 s"/>
            <property name="failRate" value="100.00%"/>
            <property name="failing" value="1"/>
            <property name="hostname" value="https://mindful-koala-mjvbtj-dev-ed.my.salesforce.com"/>
            <property name="orgId" value="00D7Q000000KLeXUAW"/>
            <property name="passRate" value="0.00%"/>
            <property name="passing" value="0"/>
            <property name="skipped" value="0"/>
            <property name="testExecutionTime" value="0.05 s"/>
            <property name="testStartTime" value="2022-06-22T17:52:38.040Z"/>
            <property name="testTotalTime" value="0.05 s"/>
            <property name="testsRan" value="1"/>
            <property name="username" value="[email protected]"/>
        </properties>
        <testcase name="testExample" classname="JsonErrorReportTest" time="0.04">
            <failure message="System.AssertException: Assertion Failed: Error, expected Key : expectedKey, generated JSON: {&quot;key&quot;: &quot;value&quot;}"><![CDATA[Class.JsonErrorReportTest.testExample: line 9, column 1]]></failure>
        </testcase>
    </testsuite>
</testsuites>

Actual result

Resulting XML:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
    <testsuite name="force.apex" timestamp="2022-06-22T17:52:38.040Z" hostname="https://mindful-koala-mjvbtj-dev-ed.my.salesforce.com" tests="1" failures="1"  errors="0"  time="0.05">
        <properties>
            <property name="commandTime" value="0.00 s"/>
            <property name="failRate" value="100.00%"/>
            <property name="failing" value="1"/>
            <property name="hostname" value="https://mindful-koala-mjvbtj-dev-ed.my.salesforce.com"/>
            <property name="orgId" value="00D7Q000000KLeXUAW"/>
            <property name="passRate" value="0.00%"/>
            <property name="passing" value="0"/>
            <property name="skipped" value="0"/>
            <property name="testExecutionTime" value="0.05 s"/>
            <property name="testStartTime" value="2022-06-22T17:52:38.040Z"/>
            <property name="testTotalTime" value="0.05 s"/>
            <property name="testsRan" value="1"/>
            <property name="username" value="[email protected]"/>
        </properties>
        <testcase name="testExample" classname="JsonErrorReportTest" time="0.04">
            <failure message="System.AssertException: Assertion Failed: Error, expected Key : expectedKey, generated JSON: {"key": "value"}"><![CDATA[Class.JsonErrorReportTest.testExample: line 9, column 1]]></failure>
        </testcase>
    </testsuite>
</testsuites>

System Information

sfdx

{
        "cliVersion": "sfdx-cli/7.155.1",
        "architecture": "darwin-arm64",
        "nodeVersion": "node-v16.15.1",
        "pluginVersions": [
                "@oclif/plugin-autocomplete 0.3.0 (core)",
                "@oclif/plugin-commands 1.3.0 (core)",
                "@oclif/plugin-help 3.3.1 (core)",
                "@oclif/plugin-not-found 1.2.6 (core)",
                "@oclif/plugin-plugins 1.10.11 (core)",
                "@oclif/plugin-update 1.5.0 (core)",
                "@oclif/plugin-warn-if-update-available 1.7.3 (core)",
                "@oclif/plugin-which 1.0.4 (core)",
                "@salesforce/sfdx-plugin-lwc-test 0.1.7 (core)",
                "alias 2.0.1 (core)",
                "apex 0.13.0 (core)",
                "auth 2.1.0 (core)",
                "community 2.0.0 (core)",
                "config 1.4.12 (core)",
                "custom-metadata 2.0.0 (core)",
                "data 2.0.3 (core)",
                "generator 2.0.1 (core)",
                "info 2.0.1 (core)",
                "limits 2.0.1 (core)",
                "org 1.13.2 (core)",
                "salesforce-alm 54.5.1 (core)",
                "schema 2.1.1 (core)",
                "sfdmu 4.14.4",
                "sfdx-cli 7.155.1 (core)",
                "sfdx-git-delta 5.3.0",
                "signups 1.1.2 (core)",
                "source 1.10.2 (core)",
                "telemetry 2.0.0 (core)",
                "templates 54.8.0 (core)",
                "trust 2.0.0 (core)",
                "user 2.0.2 (core)"
        ],
        "osVersion": "Darwin 21.5.0"
}

Jun 22 '22 18:06 jefersonchaves

Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.

Jun 22 '22 18:06 github-actions[bot]

This issue has not received a response in 60 days. It will auto-close in 7 days unless a response is posted.

Sep 21 '22 16:09 github-actions[bot]

Hi @mshanemc: is this something you can help with?

Sep 21 '22 17:09 jefersonchaves

Hey @jefersonchaves, sorry we missed this one, I'll take a look at this tomorrow. Thanks for reporting!

Sep 26 '22 14:09 cristiand391

Hey @cristiand391: thanks a lot for coming back to me. Please let me know if you need more information to reproduce this issue.

Sep 26 '22 14:09 jefersonchaves

Hello. I found this issue with invalid junit xml. For me it generated as well invalid json because of using double " inside of attributes without escaping them. This is example:

<testcase name="showsErrorIfNoFutureSprintExisting" classname="SprintPlanningCtrl_Test" time="0.02">
            <failure message="System.AssertException: Assertion Failed: (ApexPages.Message["Attempt to de-reference a null object"], ApexPages.Message["Class.CardWallCtrl.createWall: line 100, column 1
Class.CardWallCtrl.<init>: line 22, column 1
External entry point
Class.SprintPlanningCtrl_Test.showsErrorIfNoFutureSprintExisting: line 69, column 1"], ApexPages.Message["100"], ApexPages.Message["System.NullPointerException"])"><![CDATA[Class.SprintPlanningCtrl_Test.assertHasMessage: line 89, column 1
Class.SprintPlanningCtrl_Test.showsErrorIfNoFutureSprintExisting: line 74, column 1]]></failure>
        </testcase>

Inside message attribute the " are not escaped. This is version that I use: sfdx-cli/7.169.1 darwin-x64 node-v16.17.0

Sep 29 '22 06:09 gorazd-up2go

Hello @cristiand391 - Were you able to look at this? Is there more info I can provide to help you?

Sep 29 '22 07:09 jefersonchaves

This issue has been linked to a new work item: W-11831140

Sep 29 '22 18:09 git2gus[bot]

Hi @jefersonchaves, I was able to repro this with sfdx-cli/7.170.0. As @gorazd-up2go mentioned it seems the junit reporter isn't properly escaping some characters from the server response, we'll take a look at this.

Sep 29 '22 18:09 cristiand391

Great, thanks a million @cristiand391 for the update. Escaping server response should not be a novel, so please keep us posted when this might be worked, because I'm happy to test if needed.

Thanks @gorazd-up2go for reporoducing the issue.

Sep 30 '22 06:09 jefersonchaves

quick update:

the library that produces the junit test results have a PR to fix this: https://github.com/forcedotcom/salesforcedx-apex/pull/285

The team that owns it is working on fixing the unit tests and then it should be ready to include in sfdx 🙌🏼

Oct 05 '22 13:10 cristiand391

Thanks @cristiand391 : it looks like the last commit was 5 months ago. Is there more progress happening there?

Oct 05 '22 13:10 jefersonchaves

@jefersonchaves yes, I asked the team that owns it yesterday and we found that bumping test deps fixed some issues, they'll take a look at it again in the next few days

Oct 05 '22 14:10 cristiand391

Fantastic, hopefully, it can get into the next release: 1-2 weeks.

Thanks once again @cristiand391 and please let me know if you or the team needs help testing this.

Oct 05 '22 14:10 jefersonchaves

@jefersonchaves @gorazd-up2go you can try latest-rc [email protected] which includes the apex-node fix for both apex and deploy commands

see release notes: https://github.com/forcedotcom/cli/tree/main/releasenotes/sfdx#71730-oct-20-2022-stable-rc

We'll keep this issue open until the RC is promoted next Thursday.

Oct 13 '22 18:10 cristiand391

Perfect @cristiand391: I have tested both scenarios (double quotes and JSON) with Release Candidate and they worked like a charm: sfdx force:apex:test:run --classnames=TestJunit --resultformat=junit.

I have done a quick inspection with Junit Online Parser and manually.

@IsTest
public class TestJunit {

    @IsTest
    public static void generateIncorrectOutputWithDoubleQuotes() {
        throw new CustomException('This message contains "Double quotes"');
    }
    

    @isTest
    static void generateIncorrectOutputWithJson() {
        String someJsonFromIntegrationOrSomethingElse = '{"key": "value"}';

        Boolean jsonContainsKey = someJsonFromIntegrationOrSomethingElse.contains('"expectedKey"');
        System.assert(
            jsonContainsKey,
            'Error, expected Key : expectedKey, generated JSON: ' + someJsonFromIntegrationOrSomethingElse
        );
    }
    
    class CustomException extends Exception {
        
    }
}

Oct 14 '22 12:10 jefersonchaves