ios cronet test suite crash due to port server not running

[dennycd]

observing high frequent cronet test time out on both pre-submit and prod/master's grpc_basictests_objc_ios test suite. One of the following sample log captures the issue from cronet test suite crash due to python port server not running. XCTest attempts to restart the test and eventually leads to timeout being reported

• https://source.cloud.google.com/results/invocations/e2291cf0-dd86-4e84-8129-15534a947027/targets/github%2Fgrpc%2Frun_tests%2Fobjc_macos_opt_native%2Fios-test-cronettests/tests

Mon Nov  1 11:37:54 PDT 2021 - Test Case '-[CoreCronetEnd2EndTests testCancelAfterClientDone]' started.
Mon Nov  1 11:38:04 PDT 2021 - gRPC tests require a helper port server to allocate ports used
Mon Nov  1 11:38:04 PDT 2021 - during the test.
Mon Nov  1 11:38:04 PDT 2021 -
Mon Nov  1 11:38:04 PDT 2021 - This server is not currently running.
Mon Nov  1 11:38:04 PDT 2021 -
Mon Nov  1 11:38:04 PDT 2021 - To start it, run tools/run_tests/start_port_server.py
Mon Nov  1 11:38:04 PDT 2021 -
Mon Nov  1 11:38:25 PDT 2021 -
Mon Nov  1 11:38:25 PDT 2021 - Restarting after unexpected exit or crash in CoreCronetEnd2EndTests/testCancelAfterClientDone; summary will include totals from previous launches.
Mon Nov  1 11:38:25 PDT 2021 -

most sample test log doesn't capture the above crash, but instead reporting time out with the last test suite being ios cronet test,

• sample: https://source.cloud.google.com/results/invocations/ae61b11e-fcf9-4492-be80-82c981103d24/targets/grpc%2Fcore%2Fpull_request%2Fmacos%2Fgrpc_basictests_objc_ios/log

2021-11-01 14:50:10,768 START: ios-test-cronettests

2021-11-01 15:16:45,843 TIMEOUT: run_tests_objc_macos_opt_native [pid=3790, time=5402.6sec]
2021-11-01 15:16:45,856 FAILED: Some run_tests.py instances have failed.

This is followed by another script check agains the port server existence w/ exit failure

+ FAILED=true
+ ps aux
+ grep 'port_server\.py'
+ awk '{print $2}'
+ xargs kill -9
...
real	0m13.275s
user	0m0.359s
sys	0m8.198s
+ '[' true '!=' '' ']'
+ exit 1

[jtattermusch]

It might be a red herring, but note that the objc_ios tests started to fail just after https://github.com/grpc/grpc/pull/27740 was merged - so there is a high chance that this PR has caused the problem.

[dennycd]

test reverting https://github.com/grpc/grpc/pull/27897

[dennycd]

#27897 revert looks promising so far (https://fusion.corp.google.com/projectanalysis/summary/KOKORO/prod:grpc%2Fcore%2Fpull_request%2Fmacos%2Fgrpc_basictests_objc_ios) though I am still investigating the root cause for this