[BUG] Incorrect serialization failure message in Junit reporter

[Gornostalev]

Context:

• Playwright Version: [1.20.0]
• Operating System: [Windows 10]
• Node.js version: [v14.17.5]
• Browser: [All]

Code Snippet

Hello everyone 🎭✋🏻! I found incorrect junit reporter serialization work; when the test fails I expect the "failure message" field in the xml file to contain the error text, but it contains the test header:

<failure message="example.spec.ts:5:3 Check equal" type="FAILURE">

test code:

test.describe('Example', () => {

  test('Check equal', async () => {
    throw new Error('Error message');
  });
});

how it looks in json reporter:

"error": {
  "message": "Error message",
  "stack": "Error: Error message\n    at C:\\Users\\username\\Desktop\\issue\\new-project\\tests\\example.spec.ts:6:11"
},

I prepared a repository with npm scripts:

npm run testJson - for run with json reporter npm run testJunit - for run with junit reporter

I hope this makes your job easier, thanks for Playwright!❤

[dgozman]

@Gornostalev The actual failure should be a text node child of the <failure> tag:

<failure message="example.spec.ts:5:3 Check equal" type="FAILURE">
  example.test.js:5:3 › Check equal =====================================================

    Error: Error message
    ......
</failure>

Is that text missing for you?

[Gornostalev]

@dgozman, i know about this, but attribute message parsed for results in pipeline azure devops, and on screenshot field "Error message" not contain error message, it just a test title:

[dgozman]

Thank you for the screenshot, this makes sense.

Ideally, we would make the following improvements:

• Use <error> element for thrown errors, and <failure> element for expect failures.
• For errors:
  • Put error.name in type="" attribute for errors.
  • Put error message in message="" attribute.
• For failures (expects):
  • Put expect.toBe in type="" attribute.
  • Put (possibly custom) expect message in message="" attribute.