[Feature]: Provide more reliable way to map attachments with steps

[vitalets]

🚀 Feature Request

Provide more reliable way to map attachments with steps. It can be unique attachmentId field or some other approach. Please, see example below.

Motivation

This feature will give more opportunities to third-party reporters. Personally I need this for implementing Cucumber reports in playwright-bdd. As in Cucumber each attachment is mapped to a particular step although in Playwright attachments are mapped to a whole test result.

Example

Imagine the following test:

test("my test", async () => {
  await test.step('step 1', async () => {
    await test.info().attach('my attachment', { body: 'foo' });
  });
  await test.step('step 2', async () => {
    await test.info().attach('my attachment', { body: 'bar' });
  });
});

When running this test I get in onTestEnd the following (Playwright version 1.41.2): result.attachments:

[
  {
    name: 'my attachment',
    path: undefined,
    contentType: 'text/plain',
    body: <Buffer 66 6f 6f> // <- foo
  },
  {
    name: 'my attachment',
    path: undefined,
    contentType: 'text/plain',
    body: <Buffer 62 61 72> // <- bar
  }
]

result.steps:

...
{
  title: 'step 1',
  titlePath: [Function: titlePath],
  parent: undefined,
  category: 'test.step',
  startTime: 2024-02-02T14:57:40.863Z,
  duration: 1,
  steps: [
    {
      title: 'attach "my attachment"',
      titlePath: [Function: titlePath],
      parent: [Circular *1],
      category: 'attach',
      startTime: 2024-02-02T14:57:40.863Z,
      duration: 1,
      steps: [],
      location: [Object],
      [Symbol(id)]: '609f19b4a231e384f62045156c98fafe'
    }
  ],
},
{
  title: 'step 2',
  titlePath: [Function: titlePath],
  parent: undefined,
  category: 'test.step',
  startTime: 2024-02-02T14:57:40.864Z,
  duration: 0,
  steps: [
    {
      title: 'attach "my attachment"',
      titlePath: [Function: titlePath],
      parent: [Circular *1],
      category: 'attach',
      startTime: 2024-02-02T14:57:40.864Z,
      duration: 0,
      steps: [],
      location: [Object],
      [Symbol(id)]: '97c374b0a6362757b8333da5f1263208'
    }
  ],
}

In my custom reporter I want to show that:

• my attachment with text foo belongs to step 1
• my attachment with text bar belongs to step 2

Currently I don't see a reliable way to do that. I can try to match attachments names with attach "xxx" titles or play with startTimes. But it would be more reliable if we have a unique attachmentId field: result.attachments:

[
  {
    id: 'unique-id-1',  // <- notice id field
    name: 'my attachment',
    path: undefined,
    contentType: 'text/plain',
    body: <Buffer 66 6f 6f> // <- foo
  },
  {
    id: 'unique-id-2',  // <- notice id field
    name: 'my attachment',
    path: undefined,
    contentType: 'text/plain',
    body: <Buffer 62 61 72> // <- bar
  }
]

result.steps:

...
{
  title: 'step 1',
  titlePath: [Function: titlePath],
  parent: undefined,
  category: 'test.step',
  startTime: 2024-02-02T14:57:40.863Z,
  duration: 1,
  steps: [
    {
      attachmentId: 'unique-id-1',  // <- notice attachmentId field
      title: 'attach "my attachment"',
      titlePath: [Function: titlePath],
      parent: [Circular *1],
      category: 'attach',
      startTime: 2024-02-02T14:57:40.863Z,
      duration: 1,
      steps: [],
      location: [Object],
      [Symbol(id)]: '609f19b4a231e384f62045156c98fafe'
    }
  ],
},
{
  title: 'step 2',
  titlePath: [Function: titlePath],
  parent: undefined,
  category: 'test.step',
  startTime: 2024-02-02T14:57:40.864Z,
  duration: 0,
  steps: [
    {
      attachmentId: 'unique-id-2',  // <- notice attachmentId field
      title: 'attach "my attachment"',
      titlePath: [Function: titlePath],
      parent: [Circular *1],
      category: 'attach',
      startTime: 2024-02-02T14:57:40.864Z,
      duration: 0,
      steps: [],
      location: [Object],
      [Symbol(id)]: '97c374b0a6362757b8333da5f1263208'
    }
  ],
}

Pitch

Reporter data is coming from the Playwright core and can't be implemented in a third-party module.

[pavelfeldman]

We lack expressiveness of the API to reliably map attachments to steps. If steps run concurrently, we don't know which step to insert the attachment into without changing the step API and passing the attachment sink in it.

test("my test", async () => {
  await Promise.all([
    test.step('step 1', async () => {
      await test.info().attach('my attachment', { body: 'foo' });
    });
    test.step('step 2', async () => {
      await test.info().attach('my attachment', { body: 'bar' });
    });
  ]);
});

In the interim, you can compare attachments before and after the step to attribute them to the step from within onStepBegin/End. That's not perfect, but gives you a fairly good approximation.

[vitalets]

Fair enough, thanks for the idea! Will try to track attachments with onStepBegin/End. Just to clarify, in your example with parallel steps I will receive two onStepBegin events simultaneously and after some time two onStepEnds, right? So assigning added attachments will be still ambiguous for such cases?

Btw, what is the purpose of steps with attach category as they can't reliably map to actual attachment data?

[vitalets]

@pavelfeldman I've tried approach with mapping attachments in onStepBegin/End. It works fine with regular Playwright run, but breaks on merge-reports. When running merge-reports result.attachments is empty in every onStepEnd and populated only in onTestEnd.

I've created a repro here: https://github.com/vitalets/playwright-issues/tree/attachment-step-map

Do you have any suggestions how to bind attachments with steps in merge-reports run?