microsoft
[Feature]: Ability to set sharding programmatically through the config
🚀 Feature Request
We use Bazel which manages it's own sharding layer logic. Because of this, I don't have the luxury of modify the CLI command each time I want to run a given shard. I'd like to request the ability to pass a sequencer similar to how jest has https://jestjs.io/docs/configuration#testsequencer-string. This way, I can write a custom integration that allows me to orchestrate sharding based on ENV variables Bazel sets.
Example
- have a config setting called
testSequencerwhich takes a file that exports a function - this function extends some base sequencer class and then has a
sortfunctino on it that allows us to customize the sequencing - use this logic to determine what tests to run
Motivation
I'm trying to increase the parallel test running in our CI and build environments for users and I want to rule out sharding as something that we can use. Right now, it looks like we can't use it in the current state.
It looks like some variant of this may be possible today from https://github.com/microsoft/playwright/blob/a2e901e080cbf59a55b9a73d60df8bdfbb1febc6/packages/playwright/src/common/configLoader.ts#L233-L240, but it's unclear because it's not documented
You can specify it like that, does this help you?
For having a manual sharing logic it seems similar to https://github.com/microsoft/playwright/issues/17969#issuecomment-2421653419. Since you not only want to control which shard to run, you want to control the sequence and how we shard?
Yes that helps, that is what I have now. And to your point, I want to control sequencing I believe because the way https://bazel.build/reference/test-encyclopedia#test-sharding phrases it the shards need to write to a file to let bazel know how to order the sequencing. Maybe I can do that easily today through another means, like an afterAll? I'm just spitballing.
I'll look at the comment you linked as well
I want to add here that sharding is working the way I have it setup. But I will leave this open as it may be useful to have for others. Thanks for triaging!
@pavelfeldman @dgozman what do you think about this proposal to add a testSequencer configuration similar to jest’s? This would allow users of playwright to change test execution ordering (e.g. to run previously failed tests first) and how tests are split across shards (e.g. based on previous timing data…).
In fact it would unblock and allow these to be implemented in user space:
- https://github.com/microsoft/playwright/issues/17969
- https://github.com/microsoft/playwright/pull/30388
- https://github.com/microsoft/playwright/pull/30962
- https://github.com/microsoft/playwright/pull/30817
Would you be interested in a pull-request for a Test Sequencer?
Fingers crossed here that this ability will be available in version 1.5.0 🤞
@dgozman any chance to get any further with this?
It's been over a year that I created a PR which supports duration-round-robin sharding and several teams at our company (adobe) are using this via patch-package.
- https://github.com/microsoft/playwright/pull/30962
Just now, I created an updated patch for playwright 1.54.1 as your users asked for it…
+1 This would be really great to see – even just a callback function like
shardFilter: (shardNumber: number, totalShards: number, testData: SomeKindOfMetadataAboutTheTest) => boolean
This would just return whether or not a test runs in the current shard, (and I guess it no-ops if it's not fullyParallel).
Either the user takes the responsibility to ensure a test only runs in exactly one shard, or the shards could throw an exception if they detect that a test would have been run in more than one shard or zero (this could be configurable).
As mentioned in my dupe request, I have an issue where we have about 300 tests, most of which take less than 30s. But there is a suite testing a feature which is long-running, and each test will take anything from 5 to 10 minutes, causing even a sharded run to take about 20 minutes.
To be clear: the feature itself is inherently long running. I can't optimise the test, because what it is testing is an async job can be submitted (which runs on the server), and that it will complete, and show the appropriate user feedback when it does.
These tests are really hard to avoid getting clustered into the same shard – even 50 shards doesn't meaningfully help versus 15. Almost always, after about 2 to 3 minutes, every shard has finished except 1.
Hi folks! We're playing around with a new sharding feature that's useful for balancing test durations between different shards: https://github.com/microsoft/playwright/pull/38624 We'd appreciate you to try it out and leave us your thoughts in the PR.
