CumulusCI icon indicating copy to clipboard operation
CumulusCI copied to clipboard

"cci task run run_tests" generates incorrect test_results.xml format output

Open RupertBarrow opened this issue 2 years ago • 0 comments

Describe the bug

cci task run run_tests generates an invalid XML format (no <xml> header) and also an invalid JUnit format (no <testssuites> attribute)

rest_results.xml file generated :

<testsuite tests="61">
  <testcase classname="..._Test" name="..._test" time="2134" />
  <testcase classname="..._Test" name="..._Test" time="522">
    <failure type="failed" message="System.DmlException: Insert failed. ..."><![CDATA[..._Test: line 48, column 1]]></failure>
  </testcase>
  <testcase classname="..._Test" name="testIsVerified" time="10" />
</testsuite>

Expected result generated by sfdx force:apex:test:run --testlevel RunLocalTests -r junit -c :

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite errors="0" failures="0" hostname="https://xxx--yyy.sandbox.my.salesforce.com" name="force.apex" tests="3" time="35.85" timestamp="2023-01-09T10:19:27.000Z">
    <properties>
      <property name="commandTime" value="0.16 s"/>
      <property name="coveredLines" value="733"/>
      <property name="failRate" value="0%"/>
      <property name="failing" value="0"/>
      <property name="hostname" value="https://xxx-yyy.sandbox.my.salesforce.com"/>
      <property name="orgId" value="..."/>
      <property name="orgWideCoverage" value="78%"/>
      <property name="outcome" value="Successful"/>
      <property name="passRate" value="100%"/>
      <property name="passing" value="3"/>
      <property name="skipped" value="0"/>
      <property name="testExecutionTime" value="35.85 s"/>
      <property name="testRunCoverage" value="95%"/>
      <property name="testRunId" value="..."/>
      <property name="testStartTime" value="Mon Jan 09 2023 11:19:27"/>
      <property name="testTotalTime" value="35.85 s"/>
      <property name="testsRan" value="3"/>
      <property name="userId" value="..."/>
      <property name="username" value="[email protected]"/>
    </properties>
    <testcase classname="..._TEST" name="testMyController_annuler" time="3.48"/>
    <testcase classname="..._TEST" name="testMyController_error" time="3.81"/>
    <testcase classname="..._TEST" name="testMyController" time="28.56"/>
  </testsuite>
</testsuites>

Reproduction steps

  1. run cci task run run_tests
  2. save reports/test_results.xml to reports/test_results_sav.xml
  3. run sfdx force:apex:test:run --testlevel RunLocalTests -r junit -c -u <username> with the username of the CCI org used
  4. compare results

Your CumulusCI and Python versions

cci version
CumulusCI version: 3.70.0 (/Users/rupert/.local/bin/cci) Python version: 3.10.2 (/Users/rupert/.local/pipx/venvs/cumulusci/bin/python)

Operating System

MacOS 11.7.2

Windows environment

No response

CumulusCI installation method

pipx

Error Gist

No response

Additional information

No response

RupertBarrow avatar Jan 09 '23 10:01 RupertBarrow