beam_PreCommit_Python_Cron regularily failing - test_pardo_large_input flaky

[ryanthompson591]

https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/

There are a variety of tests failing, and each run has a different failure.

However, after looking through it seems that test_pardo_large_input is one of the more commonly failing tests.

Here's a link to the logs for one of the failures test_pardo_large_input

[ryanthompson591]

.add-labels P1,bug,build

[damccorm]

I've been hitting this, with many runs ending with NotImplementedError: Invalid PaneInfoEncoding: 6 (e.g. https://ci-beam.apache.org/job/beam_PreCommit_Python_Phrase/3240/testReport/apache_beam.runners.portability.fn_api_runner.fn_runner_test/FnApiRunnerTestWithGrpcAndMultiWorkers/test_pardo_large_input_5/)

I've also been seeing similarish errors with test_batch_pardo_trigger_flush

@TheNeuralBit any ideas? Is this related to https://github.com/apache/beam/issues/21643? I'm going to put up a pr to temporarily disable both

[TheNeuralBit]

Yeah it seems likely this is related to #21643. I'd suggest just tracking this as one flake rather than having the separate issue for non-cython environments

[ryanthompson591]

@damccorm after sickbaying these tests can we close this?

[TheNeuralBit]

Generally when we sickbay tests we keep the issue open to track resolving the flakiness and re-enabling the test. We've debated a lot on how to set the priority for a test in that state. My opinion: the priority should be the same as if the feature under test were broken, since we don't have evidence to the contrary.

[damccorm]

#22321 has some extra context

[kennknowles]

If this is disabled, is it masking potential problems? Or if it is just disabled and we are not worried about correctness it could be P2.