[Feature]: render iframe canvas in trace viewer

[ruifigueira]

🚀 Feature Request

As an improvement on #32248, it would be nice if canvas inside iframes could be rendered too.

I took a look on the way it's currently implemented, and I think that with some minor adjustments it can be easily achievable.

The idea is just to compute the absolute position of the canvas. Normally this would be tricky, if not impossible, due to cross origin constrains, but in snapshot rendering they all have the same origin.

Example

As an example, here's a test that opens a trace with nested canvas and patches it by injecting javascript that computes their position and clips the closest screenshot accordingly:

import { test } from '@playwright/test';

test('iframe canvas', async ({ page }) => {
  page.on('framenavigated', async (frame) => {
    await frame.evaluate(() => {
      const canvasElements = document.getElementsByTagName('canvas');
      if (canvasElements.length === 0)
        return;

      let topFrameWindow: Window = window;
      while (topFrameWindow !== topFrameWindow.parent && !topFrameWindow.location.pathname.match(/\/page@[a-z0-9]+$/))
        topFrameWindow = topFrameWindow.parent;

      const img = new Image();
      img.onload = () => {
        for (const canvas of canvasElements) {
          const context = canvas.getContext('2d')!;

          const boundingRect = canvas.getBoundingClientRect();
          let left = boundingRect.left + window.scrollX;
          let top = boundingRect.top + window.scrollY;
          let right = boundingRect.right + window.scrollX;
          let bottom = boundingRect.bottom + window.scrollY;

          let currentWindow: Window = window;

          while (currentWindow !== topFrameWindow) {
            const iframe = currentWindow.frameElement!;
            currentWindow = currentWindow.parent;

            const iframeRect = iframe.getBoundingClientRect();
            const xOffset = iframeRect.left + currentWindow.scrollX;
            const yOffset = iframeRect.top + currentWindow.scrollY;

            left += xOffset;
            top += yOffset;
            right += xOffset;
            bottom += yOffset;
          }

          const width = topFrameWindow.innerWidth;
          const height = topFrameWindow.innerHeight;

          left = left / width;
          top = top / height;
          right = right / width;
          bottom = bottom / height;

          context.drawImage(img, left * img.width, top * img.height, (right - left) * img.width, (bottom - top) * img.height, 0, 0, canvas.width, canvas.height);
        }
      };
      img.src = location.href.replace('/snapshot', '/closest-screenshot');
    }).catch(() => {});
  });
  await page.goto('https://trace.playwright.dev/?trace=https://raw.githubusercontent.com/ruifigueira/vscode-test-playwright/main/docs/assets/trace.zip');
  const actions = page.getByTestId('actions-tree').getByRole('treeitem');
  await actions.first().waitFor();
  for (const action of await actions.all()) {
    await action.click();
    await page.waitForTimeout(50);
  }
});

Notice that this computation no longer relies on __playwright_bounding_rect__.

Motivation

In my use case, I'm using playwright to test a vscode webview with a canvas, and it is always nested inside a two-level iframe structure.

I also think this solution is simpler than the current implementation while covering nested iframe canvas.

I can contribute with a PR if needed.

[ruifigueira]

Ok, maybe it's not as easy as I thought: it needs to rely on __playwright_bounding_rect__ because snapshot will adjust to window dimensions when we open it in a new tab. I'll do a few more tests later.

[Skn0tt]

Hi Rui! Thanks for looking into this. I'm not entirely sure what's the best way of implementing it, but here's a pointer: __playwright_bounding_rect__ is captured as part of the snapshot, and I believe it's the coordinates within the owning frame. If you also capture the frame bounding rect, maybe you can use that to compute the absolute coordinates?

[ruifigueira]

Yes, that was my initial approach but then I noticed __playwright_bounding_rect__ is a ratio. Nevertheless we can restore its pixels size because we also have the viewport size (basically the corresponding window.innerWidth and window.innerHeight).

I'll check its feasibility when I have some spare time

[ruifigueira]

@Skn0tt I just created #33809 with a proposal. The idea is the same, but using offsets / scroll values from the snapshots.

I had to add additional information on (i)frames and __playwright_bounding_rect__ is no longer a windows.inner{Width,Height} ratio because we must use the root window viewport instead. This may cause some regression issues, so I can give it another name.

[ruifigueira]

BTW, #33809 was failing in some snapshotter tests, I already fixed them, is now passing all tests.