chore: improve playwright test reporting on github ci

[dominikg]

let's see if we can get some useful information on ci flakiness and test results into github ui

Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

• [ ] It's really useful if your PR references an issue where it is discussed ahead of time. In many cases, features are absent for a reason. For large changes, please create an RFC: https://github.com/sveltejs/rfcs
• [ ] This message body should clearly illustrate what problems it solves.
• [ ] Ideally, include a test that fails without this PR but passes with it.

Tests

• [ ] Run the tests with pnpm test and lint the project with pnpm lint and pnpm check

Changesets

• [ ] If your PR makes a change that should be noted in one or more packages' changelogs, generate a changeset by running pnpm changeset and following the prompts. All changesets should be patch until SvelteKit 1.0

[changeset-bot[bot]]

⚠️ No Changeset found

Latest commit: 4a4a11633ac16ed02b0cdaed26a75154a64adf0a

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[dominikg]

damnit, there is a prefix interfering with the message format that gh parses https://github.com/sveltejs/kit/actions/runs/3465238448/jobs/5787732669#step:8:324

[benmccann]

how do you envision this being used? it doesn't give us anything we can use to debug. we've already got traces and screenshots for the ones that fail repeatedly, which is a lot more actionable and are the ones that are most annoying anyway

[dominikg]

this raises visibility. if the prefix wasn't there, the::warning ... string i linked would turn into a nice big box with a warning icon, title and message, and might even be repeated on the test overview page. So even with all greens we'd learn there are flakes

[benmccann]

oh, cool. I didn't realize this will show up in the github UI. I thought it'd get buried in a log somewhere. do you have an example of how this will be displayed?

[dominikg]

apologies for the screenshot from google image search but wasn't able to find a run that has one quickly

regarding the prefix: it seems to be originating from turbo and even with process.stdout.write you can't avoid it.

[dominikg]

https://github.com/vercel/turbo/issues/1363

[dominikg]

marking as draft, using test in packages/kit directly to see the potential output without turbo interfering

[dominikg]

https://github.com/sveltejs/kit/actions/runs/3465678222

looks like playwrights github reporter is a bit noisy due to us running it for multiple apps multiple times, but the flaky reporting works :partying_face:

[dominikg]

and here's a run without the noise https://github.com/sveltejs/kit/actions/runs/3465862956

[benmccann]

wow, that's awesome!