Bigtable integration tests fail to connect to emulator

[dbolduc]

• GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/e77ca00f-99ba-49d8-bc8e-997f2f116ed5;tab=detail?project=cloud-cpp-testing-resources
• Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/11897/2d7c0e227f4347bf7775900139b507f641acc1df/fedora-37-bazel-libcxx/log-e77ca00f-99ba-49d8-bc8e-997f2f116ed5.txt

Seems like everything was hunky-dory until we launched the Bigtable emulator and no tests could connect to it.

2023-08-09T19:19:11Z (+3950s): Launching Cloud Bigtable emulators in the background
2023-08-09T19:19:11Z (+3950s): Connected to the Cloud Bigtable emulator on localhost:8480
2023-08-09T19:19:12Z (+3951s): Connected to the Cloud Bigtable emulator on localhost:8490

We get this "Failed to connect to remote host: Connection refused" error for each RPC sent by the tests:

  Actual: UNAVAILABLE: Error in non-idempotent operation: failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:8480: Failed to connect to remote host: Connection refused error_info={reason=, domain=, metadata={gcloud-cpp.retry.reason=non-idempotent, gcloud-cpp.retry.function=DeleteTable}} (of type google::cloud::v2_15::Status), with a code that isn't equal to OK, but a message that is anything

The emulator was never working, which seems distinctly different than #8331

[coryan]

"never working" but the script successfully connected?

And is it likely that the first call was a DeleteTable? Oh, maybe from the code to garbage collect stale tables? Even then, shouldn't that do some kind of ListTables first?

[dbolduc]

"never working" but the script successfully connected?

Good point.

is it likely that the first call was a DeleteTable?

Pay no mind to DeleteTable. I copy/pasted a random error from the middle of the logs. All RPCs from the client library failed.

[alevenberg]

Repeated

cmake-gcs-rest-ci https://console.cloud.google.com/cloud-build/builds;region=us-east1/4a9e53b4-b67b-431f-83c5-708789a11ac2?project=936212892354

Step #3: index[4843]=UNAVAILABLE: failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:8480: Failed to connect to remote host: Connection refused

google::cloud::Status thrown: UNAVAILABLE: failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:8480: Failed to connect to remote host: Connection refused
terminate called after throwing an instance of 'google::cloud::v2_17::Status'

 Start 674: bigtable_examples_data_async_snippets
11/23 Test #650: bigtable_apply_read_latency_benchmark ................***Timeout 1500.01 sec
# Populating table tbl-2023-10-05-d51jpn3ub6sp3gcxpyqn3nalnkpmkkq5ai ..............................2023-10-05T17:46:40.615630271Z [INFO] <140116518390080> Enabled logging for gRPC calls (/workspace/google/cloud/bigtable/admin/internal/bigtable_table_admin_stub_factory.cc:59)
2023-10-05T17:46:40.617445621Z [DEBUG] <140116518390080> CreateTable() << status=google.bigtable.admin.v2.CreateTableRequest

[coryan]

Another instance:

• GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/b92b8710-e980-468d-a885-f5c69e96cf97;tab=detail?project=cloud-cpp-testing-resources
• Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/b527191fade1e34b335bac3cf71a8359ac0cf6d0/fedora-latest-bazel-coverage/log-b92b8710-e980-468d-a885-f5c69e96cf97.txt

For extra weirdness... the test fails but many requests succeed first, and then many requests in the same process also succeed. All of this is running against the emulator.

[devbww]

• GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/04ab068b-3a6c-4bdb-80c5-a28a20df1a9d;tab=detail?project=cloud-cpp-testing-resources
• Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/4a4cc34cf7683c7229acfc345a880978727df5de/fedora-latest-bazel-coverage/log-04ab068b-3a6c-4bdb-80c5-a28a20df1a9d.txt

[dbolduc]

• GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/e76a861c-74d0-4bd3-9051-44355a6b341a;tab=detail?project=cloud-cpp-testing-resources
• Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/91f2f180b071b4db92ea936b2b28729a4f904fad/fedora-latest-bazel-tsan/log-e76a861c-74d0-4bd3-9051-44355a6b341a.txt

[scotthart]

no repeats in last 90d