Chamikara Jayalath

@kileys making this a release blocker in case we have to update Python or x-lang wrappers somehow for automatic expansion service startup to not hang.

I can reproduce this locally. With grpcio 1.48.0 the "subprocess.Popen" invocation below gets stuck. https://github.com/apache/beam/blob/e5e3cb25ca4fc2e31c10eb3dbda8289c6bfc7140/sdks/python/apache_beam/utils/subprocess_server.py#L117 cmd is: ['java' '-jar' '/beam/sdks/java/io/expansion-service/build/libs/beam-sdks-java-io-expansion-service-2.42.0-SNAPSHOT.jar' '55010' '--filesToStage=/beam/sdks/java/io/expansion-service/build/libs/beam-sdks-java-io-expansion-service-2.42.0-SNAPSHOT.jar'] It works fine with grpcio 1.47.0. @tvalentyn any...

Update: we could reproduce locally. @tvalentyn is running a bisect on the grpcio repo to to identify the culprit.

Detailed instructions to reproduce using an existing Apache Beam release. #### Install Apache Beam 2.40.0 $ mkdir test_env $ python -m venv test_env/ $ . ./test_env/bin/activate $ pip install apache-beam[gcp]==2.40.0...

Yeah, I can try. Lemme know when it's ready to be tested (or I can do it when the PR is merged).

Hi @gnossen, unfortunately I'm still running into the issue both on https://github.com/grpc/grpc/pull/30473 and grpcio HEAD. @tvalentyn will you be able to check as well ? (note, that I had to...

This will unfortunately result in a breakage for our released SDKs. Also, we can very reliably reproduce this for gRPC 1.48.0 and all released Beam SDKs will automatically pick up...

Tentatively re-opening to fix the ongoing release.

Run XVR_PythonUsingJava_Dataflow PostCommit

Run XVR_Direct PostCommit