Different Playwright option, per "device"?

[jsnmrs]

trafficstars

The attention to detail you've baked into Evaluatory is inspiring. Thank you for building (and maintaining) this!

I'm interested in weaving the configuration between Playwright and the screenshot module a bit. Can you see a path to defining "devices" so that a given page's report contains both viewport-based screenshots and forcedColors + colorScheme screenshots that approximate high contrast preferences?

I can accomplish a dark text on light background with the following:

"playwrightOptions": {
  "forcedColors": "active",
  "colorScheme": "light"
}

And a light text on dark background with:

"playwrightOptions": {
  "forcedColors": "active",
  "colorScheme": "dark"
}

Those settings remain for the entirety of the Playwright session. Pulling those options into the "device" definition would require some Playwright resetting (and potentially additional execution time). I'm unsure how feasible that is, though.

In my mind, I see device configuration along these lines:

"name": "Narrow, high contrast (dark)",
"options": {
  "viewport": {
    "width": 320,
    "height": 2000
  },
  "forcedColors": "active",
  "colorScheme": "dark"
}

Ideally, I would love to corral all the screenshots together — showing various viewports and user preference outcomes per URL.

As it stands, I can run a few passes with Evaluatory, each using a different configuration to accomplish this.

[darekkay]

Thank you for the suggestion!

As it stands, I can run a few passes with Evaluatory, each using a different configuration to accomplish this.

This is indeed what I'm doing myself to evaluate both the light and dark theme variants. I can see how a combined results page would be an improvement. I would also like a solution based on the existing devices option, maybe even exposing the entire playwrightOptions per device, but I'm also not sure about the feasibility without spawning a new instance for each device. One idea would be to let the decision up to the user: if device.playwrightOptions is defined, a new instance will be spawned, otherwise Evaluatory will behave as it does today. This might be a little tricky to implement.

I would like to invest some time into this topic, but I can't promise any timeline.

[jsnmrs]

Thank you for confirming my curiosity — both your workflow and the likely need to spawn new Playwright instances to accommodate.

Still, I'm pleased that running multiple passes can generate those assets.