Patrol doesn't cooperate with FTL sharding/flank

[fylyppo]

Overview

As we have introduced test bundling, we want to use its capabilities such as test sharding.

While updating our CI workflow with Firebase Test Lab, I have tried to run sharded tests with gcloud beta firebase test android run and --num-uniform-shards option. After runs with 3 shards I need to say: It's not reliable, stable and what's most important it's not FASTER:

• Command gcloud firebase test android run without sharding takes about 8m to be finished.
• Command gcloud beta firebase test android run with 3 shards takes about 9m to be finished :disappointed:.

That's why I wanted to find better solution and it should be flank.

Issue

When I set max-test-shards: <greater than 1> it creates only one shard always. After further investigation it looks that flank sees only one test case with all tests included so it can't shard them at all.

[bartekpacia]

We should definitely investigate this strange behavior.

What baffles me is that sharding works just fine on emulator.wtf.

[prolificcoder]

Hello all, wondering if this has been figured out or there is some other work around or progress? We have a very large test suite (400+) flutter integration tests and we cannot afford to run them serially.

[prolificcoder]

This later post from @bartekpacia https://github.com/flutter/flutter/issues/101296#issuecomment-1611174497 tells that sharding is working for FTL, is that correct or is it only working for emulator.wtf

[Wreos]

Hey, Patrol team! Any updates about that? I have the same problem. As the -num-uniform-shards increases, the number of failed tests increases in FTL