Djane Luz

This is still happening, I'm using BEAM version 2.52.0 and Airflow to trigger the pipeline.

Hello @Abacn , thank you for the reply. Yes, the pipeline with 2.54.0 shows "runner v2: enabled" and the one with 2.53.0 shows "runner v2: disabled"

I'm not sure how can I open a support ticket

> For 2.54.0, can you try to disable Runner V2 (--experiments=disable_runner_v2)? I ran the pipeline with version 2.54.0 disabling the Runner V2, as suggested, and it worked.

> Is it possible for you to run your job on 2.53.0 with Runner V2 enabled? I also ran the pipeline with version 2.53.0 with the Runner V2 enabled, and...

So apparently the problem is the Runner V2 and not the Reshuffle, am I right?

> Those four error messages are from the service's point of view (after 4 it fails a batch job). I wonder if we see a crash loop or some such...

> Can you share the size of elements and the overall size of data shuffled? (I don't think any other factors could impact this transform) The input of the `Reshuffle.viaRandomKey`...

@kennknowles so maybe the problem might be on BigQueryIO write? This step currently is: ``` return response .apply("WriteToBigQuery", BigQueryIO.write() .to(tableSpec) .optimizedWrites() .withFormatFunction(new UserRunQueryResponseToTableRow()) .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_NEVER) .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_TRUNCATE));

I did like you suggested and isolated the pipeline, removing the step to Write to BigQuery, using Beam Version 2.55.1, and the pipeline succeeded!