google-cloud-cpp Integration test fails with `kDeadlineExceeded`

This happened with Bigtable, but I guess it could happen with any service.

2022-02-07T15:44:16.540779000Z [DEBUG] <0x109e0edc0> SetIamPolicy() << google.iam.v1.SetIamPolicyRequest { resource: "projects/cloud-cpp-testing-resources/instances/test-instance/tables/tbl-2022-02-07-bafh21sas3xpjrhk9b42lev1ow8gxb0jyw" policy { bindings { role: "roles/bigtable.reader" members: "serviceAccount:bigtable-test-iam-sa@cloud-cpp-testing-resources.iam.gserviceaccount.com" } } } (../../google/cloud/internal/log_wrapper.h:86)
2022-02-07T15:45:40.640512000Z [WARNING] <0x7000056db000> Failed to refresh connection. Error: DEADLINE_EXCEEDED: Connection couldn't connect before requested deadline ([../../google/cloud/bigtable/internal/common_client.cc:66](https://cs.corp.google.com/piper///google/cloud/bigtable/internal/common_client.cc?l=66))
2022-02-07T15:45:52.169534000Z [WARNING] <0x7000056db000> Failed to refresh connection. Error: DEADLINE_EXCEEDED: Connection couldn't connect before requested deadline ([../../google/cloud/bigtable/internal/common_client.cc:66](https://cs.corp.google.com/piper///google/cloud/bigtable/internal/common_client.cc?l=66))
2022-02-07T15:45:53.555493000Z [WARNING] <0x7000056db000> Failed to refresh connection. Error: DEADLINE_EXCEEDED: Connection couldn't connect before requested deadline ([../../google/cloud/bigtable/internal/common_client.cc:66](https://cs.corp.google.com/piper///google/cloud/bigtable/internal/common_client.cc?l=66))
[../../google/cloud/bigtable/admin/integration_tests/admin_iam_policy_integration_test.cc:58](https://cs.corp.google.com/piper///google/cloud/bigtable/admin/integration_tests/admin_iam_policy_integration_test.cc?l=58): Failure

I am marking this as specific to bigtable because it is the only case (currently) where we set an specific timeout. Also, this is a non-idempotent operation, so we do not retry.

Feb 07 '22 16:02 coryan

No repeats in over 90d, closing

May 23 '22 13:05 coryan

GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/94e09ac3-4450-42b0-9aaa-a72ec6afa61d;tab=detail?project=cloud-cpp-testing-resources
Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/2bb2d0203cb626553a6ae88e9490b93bd5e7685f/fedora-36-integration-daily/log-94e09ac3-4450-42b0-9aaa-a72ec6afa61d.txt

Aug 07 '22 15:08 coryan

GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/a7a3d044-c6c6-4069-9afc-9a358f31a4cb;tab=detail?project=cloud-cpp-testing-resources
Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/4e02afd6ab2cbe6a21646b220e609ded5a990de0/fedora-36-integration-daily/log-a7a3d044-c6c6-4069-9afc-9a358f31a4cb.txt

Sep 04 '22 20:09 dbolduc

GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/f33e7586-ffee-4826-b894-fe1fe413f37b;tab=detail?project=cloud-cpp-testing-resources
Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/a4d7937afee8a84841b04364544d178102ea5406/fedora-36-integration-daily/log-f33e7586-ffee-4826-b894-fe1fe413f37b.txt

Oct 12 '22 15:10 coryan

GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/a9905b0d-3843-4c00-932b-92af12e58288;tab=detail?project=cloud-cpp-testing-resources
Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/02f5f28d34e15f650ddc39351ff1c22c07452dfb/fedora-36-integration-daily/log-a9905b0d-3843-4c00-932b-92af12e58288.txt

This was in //google/cloud/bigtable/tests:instance_admin_integration_test, running against prod

Oct 21 '22 15:10 dbolduc

GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/a01c84d6-4420-4948-a528-0d88af7f5ce4;tab=detail?project=cloud-cpp-testing-resources
Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/6519a40d0d9dc897eef9262d2e4343748eb09e50/fedora-36-integration-daily/log-a01c84d6-4420-4948-a528-0d88af7f5ce4.txt

Nov 07 '22 14:11 dbolduc

GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/b4d553bd-2676-4a44-a3ee-4b3e33f46b7b;tab=detail?project=cloud-cpp-testing-resources
Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/1fbb8ebd02018d441bad97563e3711772a239cfc/fedora-36-integration-daily/log-b4d553bd-2676-4a44-a3ee-4b3e33f46b7b.txt

Nov 11 '22 13:11 coryan

GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/c7cf3ee1-f420-43be-bf85-448dcbedf312;tab=detail?project=cloud-cpp-testing-resources
Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/80feec00e286e2e677ccd6d93f60611c4a9c8d5c/fedora-36-bazel-integration-daily/log-c7cf3ee1-f420-43be-bf85-448dcbedf312.txt

Dec 04 '22 16:12 coryan

Closing in favor of #10413

Dec 09 '22 16:12 coryan