Integration tests fails after Bigtable emulator crashes

[coryan]

I have seen this pattern before, but the most damming evidence is:

EMULATOR_PID=25281 -.+.
/workspace//google/cloud/bigtable/tools/run_emulator_utils.sh: line 28: kill: (25281) - No such process

GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/debbbe77-c604-492c-8e6c-e1eb6978d643;tab=detail?project=cloud-cpp-testing-resources Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/77af2b658664c1ab9d98943e63bdc0725b837515/fedora-34-xsan/log-debbbe77-c604-492c-8e6c-e1eb6978d643.txt

[coryan]

• GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/d4bf9aa6-e11f-4bf1-bde2-8d6aeb3e98b7;tab=detail?project=cloud-cpp-testing-resources
• Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/8479/dc5d4a7a08d29c92b0854ab055f8e07f0376109e/fedora-34-tsan/log-d4bf9aa6-e11f-4bf1-bde2-8d6aeb3e98b7.txt

[dbolduc]

• GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/22a81e5a-a17f-4180-89d5-a96488bfd4a9;tab=detail?project=cloud-cpp-testing-resources
• Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/269a09a280745cf109890a247578921b7d524f11/fedora-34-coverage/log-22a81e5a-a17f-4180-89d5-a96488bfd4a9.txt

[devbww]

• GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/722e0cc2-bbde-4e89-863d-05db7d49abe8;tab=detail?project=cloud-cpp-testing-resources
• Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/c4c0510b17c8caf6c088384ef73de8ff55b4c056/fedora-35-coverage/log-722e0cc2-bbde-4e89-863d-05db7d49abe8.txt

[coryan]

No repeats in 90d, closing.

[dbolduc]

Just had a crash on a cxx-20-pr:

• GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/8c481c69-8225-48a9-ae54-bb9488c52978;tab=detail?project=cloud-cpp-testing-resources
• Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/10185/d94cb2867b202edc9b818fec01829836aa050d7e/fedora-36-cxx20-cxx20/log-8c481c69-8225-48a9-ae54-bb9488c52978.txt

And there was the following from 2022-10-20 (also captured in #10082):

• GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/7c25362c-3036-4279-8788-0abe9b9bdca0;tab=detail?project=cloud-cpp-testing-resources
• Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/c8625e063050fc406de8a5afe7913a9a1d80b082/fedora-36-cmake-gcs-rest/log-7c25362c-3036-4279-8788-0abe9b9bdca0.txt

[dbolduc]

• GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/796f25ed-1a06-4be1-b41a-df9eb87ddeb5;tab=detail?project=cloud-cpp-testing-resources
• Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/4b1a9a852840542d67ec427b36a888a39d744f00/fedora-36-cxx20-cxx20/log-796f25ed-1a06-4be1-b41a-df9eb87ddeb5.txt

cxx-20 again

[coryan]

No repeats in over 4 months. Closing.

[dbolduc]

we jinxed it

• GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/37833209-515b-443f-8d5f-3c09c3dc3f85;tab=detail?project=cloud-cpp-testing-resources
• Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/3368e3a1a5c9adbcd6e392e558740abb5d93b744/fedora-37-cxx20-cxx20/log-37833209-515b-443f-8d5f-3c09c3dc3f85.txt

[coryan]

• GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/8a5a5bed-bc0b-4feb-8298-0ddb49968003;tab=detail?project=cloud-cpp-testing-resources
• Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/beb4694940ed246d2d8fc5953ce2f531daf74fdd/fedora-37-bazel-libcxx/log-8a5a5bed-bc0b-4feb-8298-0ddb49968003.txt

[coryan]

• GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/a16193a6-6d88-4b91-bcc1-6c4da060b6a6;tab=detail?project=cloud-cpp-testing-resources
• Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/6c20b1b0070bf9807649cae237aecd95c8a91b2e/fedora-37-cxx20-cxx20/log-a16193a6-6d88-4b91-bcc1-6c4da060b6a6.txt

[coryan]

• GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/06ff5ac6-70f4-4dd7-9922-1b19df8f4a4c;tab=detail?project=cloud-cpp-testing-resources
• Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/d9a2924ddbb8d3bc01b8bd60bd9adcd5d0e8d5d3/fedora-37-cxx14-cxx14/log-06ff5ac6-70f4-4dd7-9922-1b19df8f4a4c.txt

[coryan]

• GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/14a990a0-a7a3-4e06-98c8-e39f4311d088;tab=detail?project=cloud-cpp-testing-resources
• Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/c6c3f0116742b534dd97cb551083b9b7f09d41b0/fedora-37-cmake-clang-tidy/log-14a990a0-a7a3-4e06-98c8-e39f4311d088.txt

[coryan]

Snooze.

[coryan]

• GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/42004e0d-48a2-408a-a511-411093be823a;tab=detail?project=cloud-cpp-testing-resources
• Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/12816/953896bef3582ea33a29f1810b59fdea1338e643/fedora-latest-cxx14-cxx14/log-42004e0d-48a2-408a-a511-411093be823a.txt

[devbww]

Perhaps we shouldn't be surprised that the emulator can crash given the size of that MutateRowsRequest in the log?

[coryan]

We should at lease reduce the size to the minimum needed to exercise the functionality we want.

[alevenberg]

FAILED: //google/cloud/bigtable/tests:table_sample_rows_integration_test (Summary)
      /h/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/execroot/google_cloud_cpp/bazel-out/k8-fastbuild/testlogs/google/cloud/bigtable/tests/table_sample_rows_integration_test/test.log
      /h/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/execroot/google_cloud_cpp/bazel-out/k8-fastbuild/testlogs/google/cloud/bigtable/tests/table_sample_rows_integration_test/test_attempts/attempt_1.log
      /h/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/execroot/google_cloud_cpp/bazel-out/k8-fastbuild/testlogs/google/cloud/bigtable/tests/table_sample_rows_integration_test/test_attempts/attempt_2.log
INFO: From Testing //google/cloud/bigtable/tests:table_sample_rows_integration_test:
==================== Test output for //google/cloud/bigtable/tests:table_sample_rows_integration_test:
Note: Randomizing tests' orders with a seed of 71034 .
[==========] Running 4 tests from 1 test suite.
[----------] Global test environment set-up.
google/cloud/bigtable/testing/table_integration_test.cc:90: Failure
Value of: TableAdminClient().CreateTable( bigtable::InstanceName(project_id_, instance_id_), table_id_, std::move(t))
Expected: code is equal to OK and message is anything
  Actual: 168-byte object <90-80 4A-02 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 ... 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00>, whose status is UNAVAILABLE: Error in non-idempotent operation: failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:8480: Failed to connect to remote host: Connection refused error_info={reason=, domain=, metadata={gcloud-cpp.retry.reason=non-idempotent, gcloud-cpp.retry.function=CreateTable, gcloud-cpp.retry.original-message=failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:8480: Failed to connect to remote host: Connection refused}}, with a code that isn't equal to OK, but a message that is anythin

[alevenberg]

Build FAILURE: ubsan-ci https://console.cloud.google.com/cloud-build/builds;region=us-east1/f63204ee-8ae4-49af-a67f-cf16a990c7ca?project=936212892354

[alevenberg]

https://ff931f6f2be3260946b05593077ad856417dea46ec2ce691e3facff-apidata.googleusercontent.com/download/storage/v1/b/cloud-cpp-community-publiclogs/o/logs%2Fgoogle-cloud-cpp%2Fmain%2Fa988f3e39ca5134b6966578c3db5da07e1147156%2Ffedora-latest-bazel-ubsan-default%2Flog-f63204ee-8ae4-49af-a67f-cf16a990c7ca.txt?jk=ATxpoHdHTuNzIt2mSYitpwr1r4XpGkxX7u8OWXmzzX9ZDSjSDtgUUqDbfOKM3Ny8Iwe6QP16bfOEFcmaorroNHPXNJKYCcOWOI_KLW4fdAkHHDYpwElCoWr4F0LZmSzxKR0wX3EjMohbE8jYcOny3BU_B9Ya_UTexxiUPe4wN8SlbCgotJUYYLxShEaMy7toTgfIZihy4evrb2chINOqkgYk9GV4CJUOvQi1n4JKkv-yzInPuxU7zZ39DK6wpuNBtpJAd0yIs1Rf4CWbKhY1Z9ysU4NHgQxai5SkI4D2_L8S5-3lEE-2QdxARijX6xIevdiiEC37txa7rA1F1u_ztVY0Wm8IInN-VVGVZh8QohsN3TaKAUXdZPlz1C5eH2lBx7h_JwGs-aedzRgxBxbfDVepbz4AIZrM0B3cRDMxnGTOi9q2mVFVWYtHdllze_56QLQ1XkB8K-n_P7WSxYvmzIi01RIBNeaNf5t_jCElFHMlsvI7XOUhnj2GXPVDtgVpkIQHEfUa_h_S7CWeWGTgcq2hospofn56WTtX8Kup9MhcbloMxJyYTrz6kH5vc6AkZPo_lZGOYCKS87SA-0r8DbMqhVEHktyVW1hyo7jo45hrtRjtD882OQPNzQW5NEJvjg8-RK7MOugzS5-3LWiLPNaB2Vtsm6UYkFoFKSX4UxxVC223o_L6RSuDHh8gKPoPmGV1h2dinOh6t1cci5Rtrq_nefXuh25iTsqHW9YksDo_M_lACB6OJC3lKcuer7wm262kfOjIMpoqw_lCgLocqpemibXoAaBiuJsyBJRXgWu2EQnT92zW43D-cspIxkyOIm40nzntu0bQVzVJHLwhyhYDcZ38yJu_SQzCHwyOzwOEXWVVxOvC5NBZ5xQAHCEGU696KhyqxrdIByW3zLV35MVwJAg0qeMGyHb9p89qxaDpXrAH7uFg5JRYo20W4-1ut0Jte5rMfg7bFYDg4Y-AHUaquw8-oHpDJnVCUlgQUNnxt8v9oIAyRRQkV_Su3wWig5qyVKGf5TaQu5c3BK4OoSxk1-J9UT0kgsXQWV33lcC0GUvFbr740tgHh3DpJDX_oiKnL9Oojysj_X3XdWKyDF1-G-cOaikSWKsNUYJkENm07xQ8U9sm3_KxEba1qPa0CeYIkWKu5-VporSznHYUa6CxJMxqXe9GXqKHOs-8J8PpEroWRDqZR8v0aSkQWGAbU3UVa4S4TJpfFDQrVLeM8jG8hOSqWCWXLdSyiNA4ECiff8FeHsyyrlFX6d649U5rgzUX5Ssd29lVerw08TSF9FpQ6_7sQFw9-936zxAkwUKTOnOJqGPeRFELUxapXDYMqkymTJQSyfA0ltcuyNwjxsM6x1v4C03mOs9YMUtdHUd7yPoQ_TmOe4JN&isca=1

[scotthart]

Closing, no occurrences in 90d.