[Feature] Mouse helper (tracking mouse position, clicks, drag & drops, selections)

[thernstig]

It would be nice to natively have support for a mouse helper. So if running in headful mode, or taking a screenshot, or video recording, one would see where the mouse is, text selections, drag & drops, and where it clicks during a failure.

@aslushnikov already had one in Puppeteer but it required installing it, but maybe it could be default-installed on the browserContext or page instead if wanted, or even for the browser. Having it out of the box would improve usability for all users. Especially during debugging.

https://github.com/puppeteer/puppeteer/issues/4378#issuecomment-499726973

[aslushnikov]

This will be addressed by https://github.com/microsoft/playwright/pull/3526#pullrequestreview-472161209

[LanderBeeuwsaert]

Yes, in our test cases a lot of the time it's difficult to know at exactly what line playwright failed. Having visual confirmation; most importantly in the video if & where mouse clicks were performed will for some issues certainly help us to identify them. In cypress when you hover over an action in the action bar & the action is a click, you see the click on the screenshot for that moment, which is quite helpful.

[aslushnikov]

This should be now addressed with a much more advanced ui - please check out the new playwright inspector: https://playwright.dev/docs/inspector/

I'll close this bug now. We continue working on making inspector UI nicer and more helpful, so if you have any issues or suggestions for it, please file separate issues!

[thernstig]

@aslushnikov Maybe I am missing something, but how does this work if I execute a page.screenshot() as a custom handling when failures happen? Or if I use the record functionality, to see the mouse move?

I.e. I want this to use this in CI when a test case fails, I take a screenshot, and see where it failed. Without having to re-run with the inspector.

[LanderBeeuwsaert]

yeah, that would also be our preferred usage. We want to run it during our CI runs. We found out that +- 40% of the work is building the tests; but +-60% of the time we spend on hardening them when 1 in 20 runs a command fails because for that run it behaves a bit different because the CPU was slower or whatever. It's there that we would like to gain time.

[aslushnikov]

@thernstig @LanderBeeuwsaert good points! Re-opening then

[thernstig]

@aslushnikov great :) No idea of how you achieved this with the inspector, but hopefully you can re-use some of that code for also this scenario we described i.e. in CI.

[klhex]

Interestingly, there is already such a mouse helper available separately, originally made for Puppeteer it seems: https://gist.github.com/aslushnikov/94108a4094532c7752135c42e12a00eb

(Doesn't compile on my side because page.evaluateOnNewDocument does not exist in Playwright. Oh, when you rename it to page.addInitScript then it works...)

However, of course it would be really nice and helpful, to have such a feature integrated in Playwright, including the possibility to activate it as needed via API (e.g. page.debug( { mousehelper: true } )) so that one can use it in an environment where no manual interaction is possible (i.e. CI/CD).

@thernstig The PW inspector does highlight mouse positions upon actions but no mouse movements, AFAIK. --- cut here --- --- cut here ---

[thernstig]

@klhex that was linked from the original post, and if you look closely you will notice that it is @aslushnikov who has written it 😛

[klhex]

@thernstig Right, I didn't notice that it was already linked in the first comment. Guess it's time to call it a day. ;) (However, I did notice that it was written by aslushnikov. :)

[ChrisLasar]

Any news on this? Would love to see this feature. The maybe related pull request (https://github.com/microsoft/playwright/pull/3526#pullrequestreview-472161209) is still closed. Would like to see this feature to automatize generation of small screen videos for tutorials. Any other ideas and hints on how to do this?

[lourd]

Linking to related issues #6737 and #5816. These are critical for my team for using Playwright to test our WebGL app, https://spatial.io

[klhex]

This would be especially useful for CI/CD environments! In my case, I am running all my tests in the background (CI/CD) in headed mode with screen recording enabled by default, incl. automatic upload of the recordings to S3. With a mouse helper that I would also activate by default for all my tests, I would be able to detect issues related to flakiness, most prominent example being that PW confirms it has successfully clicked on "Next" button but nevertheless the application is not navigating to the next page, as it normally should. If I run the test again, usually the issue does not reappear (flake) and therefore I cannot use the PW trace viewer to analyze the issue further. Currently, the only alternative would be to enable the trace viewer by default for all test executions but this would be overkill since such flake issues only occur rarely and having it always activated would slow down the tests quite a bit...

[baptisteArno]

+1, that would be so helpful!

[DmitryLobachov]

+1

[ivelin]

Seems like using Tracing with Action Snapshot turned on does the job: https://playwright.dev/docs/trace-viewer#snapshots

[cenfun]

You can simulate a mouse with mouse-helper

npm i mouse-helper

Usage

const { chromium } = require('playwright');

const browser = await chromium.launch();

const context = await browser.newContext();
await context.addInitScript({
    path: './node_modules/mouse-helper/dist/mouse-helper.js'
});

const page = await context.newPage();
await page.goto("your page url");

await page.evaluate(() => {
     window['mouse-helper']();
});

await page.mouse.move(20, 50);
await page.screenshot({
    path: "your screenshot path"
});

await page.mouse.down();
await page.screenshot({
    path: "your screenshot path"
});

await page.mouse.up();

[storenth]

I think trace (and its viewer) doesn't cover all dev needs. Looks like some helper need to draw what is mouse/keyboard doing as part of the video object. Second vector is a Chrome extension to highlight mouse events.

[fjcomas]

Is there a way to add the red circle (tracer) to the videos? it does it for the Actions but when a video is created it loses it and it is hard to follow.

[ajscott-hcs]

I'd like to add to this conversation where it would be nice to have the ability to have the cursor within the test, for screenshot or video purposes - that's not in a debug or trace viewer context.

[edit[ There was a PR that was closed due to the introduction of pwdebug=1, but this doesn't cover the use cases that I require, as I need the cursor as part of the test outputs, as screenshots and videos.

Possible test cases for a feature like this would be to ensure that the cursor style changes over certain elements, as well as ensuring hover cases are visually and correctly applying for browser state.

[hriverahdez]

I tried the solution posted above by @cenfun using the mouse-helper and that indeed helps. However, a solution like this also adds the mouse cursor itself to the snapshot, so a slightly different cursor position leads to a failed test (ofc we could also add a pixel diff threshold).

It would be nice however if Playwright itself provided such option which would get excluded from screenshot matching. Our use cases is that we have a webgl app and we want to fire off actions based on canvas positions and run snapshot assertions and in many cases having a visible mouse cursor helps to sanity-check the snapshot and for easier debugging