google-cloud-cpp icon indicating copy to clipboard operation
google-cloud-cpp copied to clipboard
verified publisher icon googleapis

Make integration tests more tolerant of transient failures

Open coryan opened this issue 3 years ago • 30 comments

Our integration tests often perform "non-idempotent" operations, but these can fail even if our code is working fine. A typical test may have this structure:

  auto create = client.CreateFoo(....);
  ASSERT_STATUS_OK(create);

  auto update = client.UpdateFoo(...);
  ASSERT_STATUS_OK(update);

  auto del = client.DeleteFoo(...);
  ASSERT_STATUS_OK(del);

None of those operations are retried by default, as the operations are non-idempotent. I think we could rewirte the tests as follows:

  auto create = client.CreateFoo(... Options{}.set<IdempotencyPolicy>(RetryEverything()));
  ASSERT_THAT(create, StatusIs(AnyOf(StatusCode::kOk, StatusCode::kAlreadyExists)));

Or if we don't want to retry the operation, we can write it as:

  auto create = client.CreateFoo(...);
  if (IsTransient(...)) GTEST_SKIP() << create.status();

This probably requires a mini-design doc, and will require breaking down the work per test.

For now I am going to consolidate all existing "transient error on non-idempotent operation" flakes to this one.

coryan avatar Dec 09 '22 16:12 coryan

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/58c92e1b-75be-4b56-ba5c-9ee196d61462;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/f601caa1d9fe320d16f7cbf9fb0ce039aad95d24/fedora-36-bazel-integration-daily/log-58c92e1b-75be-4b56-ba5c-9ee196d61462.txt

coryan avatar Dec 09 '22 16:12 coryan

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/7739ac75-4eff-489a-90cd-a43fbd790fb5;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/e69ab2388d536ae42163086417c2bafeed0f1156/fedora-36-bazel-integration-daily/log-7739ac75-4eff-489a-90cd-a43fbd790fb5.txt

coryan avatar Dec 10 '22 16:12 coryan

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/048a20be-6ffd-4762-b07c-6194ef5ebb00;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/8417925c4ad384a5f6bb4c2e7692b3c08f5b2bf7/fedora-36-bazel-integration-daily/log-048a20be-6ffd-4762-b07c-6194ef5ebb00.txt

Same as #9870

dbolduc avatar Dec 27 '22 04:12 dbolduc

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/fe0f6ecb-5053-4276-8116-a59c9655e228;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/afc873acb63102303ad6e94f9a9ac5d4aadc5d06/fedora-36-bazel-integration-daily/log-fe0f6ecb-5053-4276-8116-a59c9655e228.txt

coryan avatar Jan 03 '23 13:01 coryan

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/c3736819-3111-42f4-b269-8791fc8c3648;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/9c56fb561b89e5809f6f4a049baf354ecd79d9fc/fedora-36-bazel-integration-daily/log-c3736819-3111-42f4-b269-8791fc8c3648.txt

coryan avatar Jan 06 '23 13:01 coryan

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/843cc614-1555-45d5-8445-01c33c23a79b;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/10557/646d561deec92dd01ef9618f23d73b79f19a88fd/fedora-36-cmake-quickstart-production/log-843cc614-1555-45d5-8445-01c33c23a79b.txt

optimization quickstart

Step #3:   google::cloud::Status thrown: UNAVAILABLE: Error in non-idempotent
Step #3:   operation BatchOptimizeTours: The service is currently unavailable.

dbolduc avatar Jan 12 '23 02:01 dbolduc

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/f18e5971-b455-4832-8584-34b7f9fca181;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/10557/646d561deec92dd01ef9618f23d73b79f19a88fd/fedora-36-cmake-integration-production/log-f18e5971-b455-4832-8584-34b7f9fca181.txt

bad gateway 502

[ RUN      ] RestClientIntegrationTest.AnythingPutJsonContentTypeTwoSpans
google/cloud/internal/curl_rest_client_integration_test.cc:72: Failure
Value of: response->StatusCode()
Expected: is equal to 200
  Actual: 502 (of type google::cloud::rest_internal::v2_7_0::HttpStatusCode)

dbolduc avatar Jan 12 '23 02:01 dbolduc

(x3) Biquery quickstart internal error. (similar to #9728)

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/1662b942-e150-4e24-a03c-59a2d1d2b9de;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/10752/b773eefa8d485ff0bf646c8cc95cef504e397935/demo-fedora-demo-install/log-1662b942-e150-4e24-a03c-59a2d1d2b9de.txt

dbolduc avatar Feb 06 '23 04:02 dbolduc

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/4c463770-095d-4a3a-94db-38614f9c3441;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/73494dd37ba767805da0e2b0e845fdc04db372c9/fedora-37-bazel-tsan/log-4c463770-095d-4a3a-94db-38614f9c3441.txt

pubsub iam samples unavailable

Running GetSubscriptionPolicy() sample
google::cloud::Status thrown: UNAVAILABLE: Error in non-idempotent operation GetIamPolicy: Policy checks are unavailable.

dbolduc avatar Feb 21 '23 17:02 dbolduc

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/9dcc5e62-58a6-442e-aaeb-8e4584588971;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/c6cca3acd7c7d24c4f54db8eca4ba9886fffa61b/fedora-37-cmake-quickstart-production/log-9dcc5e62-58a6-442e-aaeb-8e4584588971.txt

optimization quickstart

dbolduc avatar Feb 22 '23 05:02 dbolduc

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/62b46129-ed84-45d7-baa0-0c4060e10c2a;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/99c116535257067ffa648fa4c4e6106a9ad6048c/fedora-37-bazel-xsan/log-62b46129-ed84-45d7-baa0-0c4060e10c2a.txt
=================== Test output for //google/cloud/iam:samples_iam_credentials_samples:
google::cloud::Status thrown: UNAVAILABLE: Error in non-idempotent operation GenerateAccessToken: 502:Bad Gateway

dbolduc avatar Feb 22 '23 23:02 dbolduc

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/9bdbe8d1-8b97-407f-9202-02e46e3722ac;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/84b3a304206ba5ea43220688e363a61a6236fbf0/fedora-37-bazel-libcxx/log-9bdbe8d1-8b97-407f-9202-02e46e3722ac.txt

coryan avatar Feb 27 '23 16:02 coryan

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/6d05684f-4c39-4411-8c7d-43f238397a94;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/84b3a304206ba5ea43220688e363a61a6236fbf0/fedora-37-bazel-gcs-grpc/log-6d05684f-4c39-4411-8c7d-43f238397a94.txt

But this is easy to fix.

coryan avatar Feb 27 '23 16:02 coryan

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/692cad80-c344-4b50-8eaf-95406213021e;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/63fd5d00d4305413f9abdb153c1c70b6399a906a/fedora-37-bazel-integration-daily/log-692cad80-c344-4b50-8eaf-95406213021e.txt

UNAVAILABLE for bigtable ReadModifyWriteRow()

dbolduc avatar Mar 22 '23 15:03 dbolduc

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/26609470-aeff-40ef-a132-7dc9f3170bf8;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/7811fc4e7319fcdb92cd752db84150455653e71f/fedora-37-bazel-coverage/log-26609470-aeff-40ef-a132-7dc9f3170bf8.txt
2023-03-23T05:30:33.863986555Z [DEBUG] <140451751680640> CreateSubscription() >> status=UNAVAILABLE: The service is currently unavailable. (./google/cloud/internal/log_wrapper.h:83)
terminate called after throwing an instance of 'google::cloud::v2_9_0::RuntimeStatusError'
  what():  UNAVAILABLE: Retry policy exhausted in CreateSubscription: The service is currently unavailable.

pubsub retry loop was exhausted. The default timeout is 1 min. The "fix" is probably to increase that in the test. https://github.com/googleapis/google-cloud-cpp/blob/7811fc4e7319fcdb92cd752db84150455653e71f/google/cloud/pubsub/internal/defaults.cc#L55-L58

dbolduc avatar Mar 23 '23 05:03 dbolduc

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/bde6f00f-ce54-4915-8ae1-a573e310cc38;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/b3702428e25411c1ce9f2530c3f59d91c588f3c4/fedora-37-bazel-integration-daily/log-bde6f00f-ce54-4915-8ae1-a573e310cc38.txt

Bigtable receives kDeadlineExceeded while applying mutations to set up a test.

[ RUN      ] MutationIntegrationTest.DeleteFromColumnForReversedTimestampRangeTest
google/cloud/bigtable/testing/table_integration_test.cc:242: Failure
Value of: failures
Expected: is empty
  Actual: { 16-byte object <C0-DB 87-00 00-00 00-00 00-00 00-00 00-00 00-00> }, whose size is 1
google/cloud/bigtable/testing/table_integration_test.cc:263: Failure

Step #3: 2023-04-18T07:18:17.344696255Z [DEBUG] <139973904995776> Read(7)() >> DEADLINE_EXCEEDED: Deadline expired before operation could complete. (./google/cloud/internal/streaming_read_rpc_logging.h:60)

dbolduc avatar Apr 18 '23 16:04 dbolduc

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/d05fe8bb-47b0-4a02-a227-cb9c15b8dc4b;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/fb102730887e06913a51bff9cb414f1690be582b/fedora-37-bazel-libcxx/log-d05fe8bb-47b0-4a02-a227-cb9c15b8dc4b.txt

coryan avatar Apr 24 '23 12:04 coryan

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/be8e1a9c-e3dc-48ec-a3a2-0129f25cb435;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/fb102730887e06913a51bff9cb414f1690be582b/fedora-37-bazel-integration-daily/log-be8e1a9c-e3dc-48ec-a3a2-0129f25cb435.txt

coryan avatar Apr 24 '23 12:04 coryan

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/3defdec7-c37b-4d81-ad63-ce9b88dcafef;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/1d1ad99da69cad2070622611e8bd650caa1af62c/fedora-37-bazel-integration-daily/log-3defdec7-c37b-4d81-ad63-ce9b88dcafef.txt

This may be easy to fix:

google/cloud/bigtable/admin/integration_tests/instance_admin_integration_test.cc:85: Failure
Expected equality of these values:
  0
  resp.failed_locations_size()
    Which is: 3

That indicates that 3 zones were unavailable during the test, but we really do not care unless then are the zones we want to use.

coryan avatar Apr 26 '23 21:04 coryan

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/88b5eeb0-a49e-49da-b39b-ce87a6c99250;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/035cde3b798ccde222ea139cabbc8a2225131629/fedora-37-bazel-gcs-grpc/log-88b5eeb0-a49e-49da-b39b-ce87a6c99250.txt

This is probably easy too:

Running ChangeRetryPolicy()
google::cloud::Status thrown: UNAVAILABLE: Retry policy exhausted InsertObjectMedia: We encountered an internal error. Please try again. error_info={reason=backendError, domain=global, metadata={http_status_code=503}}
2023-04-26T16:06:29.648032607Z [INFO] <140010760370816> InsertObjectMedia() << InsertObjectMediaRequest={bucket_name=gcs-grpc-team-cloud-cpp-testing-examples-2023-04-26_eoz9dc9hln1, object_name=object-BxYRQs0z26PBvl41vpHCuVkJnvpJCUFMKR8WLSYHMZUzmTuHyX7nkrBxc6iD391ODv9ntDgUxcNjgwy0xcKIPGVgo0Q69d9YXdvvXiiegAYZ76LsbsNSDpxsUyz5CYYY, disable-md5-hash=1, contents=The quick brown fox jumps over the lazy dog} (google/cloud/storage/internal/logging_client.cc:44)
2023-04-26T16:06:39.544748235Z [INFO] <140010760370816> InsertObjectMedia() >> status={UNAVAILABLE: We encountered an internal error. Please try again. error_info={reason=backendError, domain=global, metadata={http_status_code=503}}} (google/cloud/storage/internal/logging_client.cc:49)

Looks like the sample should increase (instead of dramatically decreasing) the retry policy:

https://github.com/googleapis/google-cloud-cpp/blob/347414d962ce6cd29c462c71a2dea5d3133ff9de/google/cloud/storage/examples/storage_client_per_operation_samples.cc#L94

coryan avatar Apr 27 '23 00:04 coryan

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/85a90276-c783-4f7e-8391-d7444e3dd759;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/8644724f094206a1f7b5a918906ea2dd271f8a66/fedora-37-bazel-coverage/log-85a90276-c783-4f7e-8391-d7444e3dd759.txt
[ RUN      ] UnifiedRestCredentialsIntegrationTest.ServiceAccountCredentials
google/cloud/internal/unified_rest_credentials_integration_test.cc:68: Failure
Value of: parsed.contains("kind")
  Actual: false
Expected: true
parsed={"error":{"code":503,"errors":[{"domain":"global","message":"Error encountered during execution. Retrying may solve the problem.","reason":"backendError"}],"message":"Error encountered during execution. Retrying may solve the problem.","status":"UNAVAILABLE"}}
google/cloud/internal/unified_rest_credentials_integration_test.cc:162: Failure
Expected: MakeBigQueryRpcCall(Options{}.set<UnifiedCredentialsOption>( MakeServiceAccountCredentials(contents))) doesn't generate new fatal failures in the current thread.
  Actual: it does.
[  FAILED  ] UnifiedRestCredentialsIntegrationTest.ServiceAccountCredentials (10807 ms)

devbww avatar May 05 '23 23:05 devbww

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/09121dfa-15c3-4b80-be35-3a7401147512;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/6d5e7f9802dbc4c4305e5e8816fb3e7f5099dc0d/fedora-msan-msan/log-09121dfa-15c3-4b80-be35-3a7401147512.txt

same rest credentials flake as above.

dbolduc avatar May 17 '23 19:05 dbolduc

Bigquery quickstart internal error. (similar to https://github.com/googleapis/google-cloud-cpp/issues/9728)

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/a951377c-0d4c-4db2-8923-2e118b0b177f;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/c71213576be3908f20b1f8c080dfb5306fa9e646/demo-centos-7-demo-install/log-a951377c-0d4c-4db2-8923-2e118b0b177f.txt

dbolduc avatar Jun 13 '23 15:06 dbolduc

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/16e026e5-c128-425a-88e9-92a2d9632f6a;tab=detail?project=cloud-cpp-testing-resources

  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/24a8043c44feaf7ef1d8fc1410308c5706a69125/fedora-37-cmake-quickstart-production/log-16e026e5-c128-425a-88e9-92a2d9632f6a.txt

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/15b588f3-9daa-49ea-9f0c-cf28acaf76c9;tab=detail?project=cloud-cpp-testing-resources

  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/24a8043c44feaf7ef1d8fc1410308c5706a69125/fedora-37-cmake-quickstart-production/log-15b588f3-9daa-49ea-9f0c-cf28acaf76c9.txt

coryan avatar Jun 27 '23 20:06 coryan

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/0d2abaaa-6904-4674-bcc4-920352f7066e;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/391e998abb6401a27e70c61a20e733f4da875a45/fedora-37-bazel-tsan/log-0d2abaaa-6904-4674-bcc4-920352f7066e.txt

This is a timeout trying to upload to GCS using a signed URL. It seems the retry loop does not even get to run, because the policy expires on the first error (it takes 2 minutes to timeout). We should consider shorter timeouts (to make more attempts) or retrying based on error count by default.

2023-06-29T00:13:56.047326040Z [INFO] <140370954560128> Enabling logging for rpc-streams (google/cloud/storage/client_options.cc:222)
2023-06-29T00:13:56.047994766Z [INFO] <140370954560128> InsertObjectMedia() << InsertObjectMediaRequest={bucket_name=gcs-grpc-team-cloud-cpp-testing-bucket, object_name=ob-NVT5MaCOvNqJvO9CwLI7SyjVBpBgeTlOt4krFUoEDYpvPCV15aFf3ZLGDc6CfPhOlUn2kgVnWzyNbOuKyZeHbJAIG5uJ2ulxS2qVuGf98Aiyjpvkft1VHU9CRZhFfq1C.txt, disable-md5-hash=1, ifGenerationMatch=0, contents=Lorem ipsum dolor sit amet, consectetur adipiscing.elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut e...<truncated>...} (google/cloud/storage/internal/logging_client.cc:44)
2023-06-29T00:15:56.087931208Z [INFO] <140370954560128> InsertObjectMedia() >> status={DEADLINE_EXCEEDED: PerformWork() - CURL error [28]=Timeout was reached} (google/cloud/storage/internal/logging_client.cc:49)
2023-06-29T00:15:56.089510009Z [INFO] <140370954560128> Enabling logging for raw-client (google/cloud/storage/client_options.cc:222)

coryan avatar Jun 29 '23 00:06 coryan

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/4eb72b27-9478-4f19-a8f5-e36bd1b4a5df;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/816214b7cedd54a087ea033995bacb6a2b54b438/fedora-37-bazel-tsan/log-4eb72b27-9478-4f19-a8f5-e36bd1b4a5df.txt

There is some kind of retry already going on here, which failed after 120s.

coryan avatar Jul 05 '23 15:07 coryan

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/46452c2a-8670-4cde-a6d3-6837056d4688;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/6cb9140fbc9273cf8445059ea3a9aef6aba5687e/fedora-latest-bazel-coverage/log-46452c2a-8670-4cde-a6d3-6837056d4688.txt

Maybe we should skip the production-only samples and tests in the coverage build. We will be a bit less informed about coverage, but the noise will be lower. I convinced myself.

coryan avatar Oct 12 '23 23:10 coryan

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/7c1c837c-4d62-4b10-acb6-59350c71b10a;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/6b216de62b0f2adef72073ff9ffbeac582aeb66c/fedora-latest-bazel-coverage/log-7c1c837c-4d62-4b10-acb6-59350c71b10a.txt

Mitigated by: https://github.com/googleapis/google-cloud-cpp/pull/13132

coryan avatar Nov 14 '23 22:11 coryan

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/ff102916-4721-4096-a013-58e1327adae8;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/cb500d59eb0fd16b9ccd18d503ff9f15f0a36730/fedora-latest-cmake-quickstart-production/log-ff102916-4721-4096-a013-58e1327adae8.txt

Quota exhausted ... possibly because we have too many builds, all of them try to find any builds to cancel?

google::cloud::Status thrown: RESOURCE_EXHAUSTED: Permanent error, with a
  last message of Quota exceeded for quota metric 'Other API requests' and
  limit 'Other API requests per minute' of service
  'cloudbuild.googleapis.com' for consumer 'project_number:936212892354'.
  error_info={reason=RATE_LIMIT_EXCEEDED, domain=googleapis.com,
  metadata={gcloud-cpp.retry.reason=permanent-error,
  gcloud-cpp.retry.original-message=Quota exceeded for quota metric 'Other
  API requests' and limit 'Other API requests per minute' of service
  'cloudbuild.googleapis.com' for consumer 'project_number:936212892354'.,
  quota_location=global, gcloud-cpp.retry.function=ListBuilds,
  service=cloudbuild.googleapis.com, quota_limit_value=600,
  consumer=projects/936212892354,
  quota_limit=AllElseRequestsPerMinutePerProject,
  quota_metric=cloudbuild.googleapis.com/all_else_requests}}

coryan avatar Apr 29 '24 16:04 coryan

  • GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/1e4dde2f-0280-48fb-a945-5fa40b8a3afa;tab=detail?project=cloud-cpp-testing-resources
  • Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/aab2de313def893cd0ca8935a24c098614e0c7e1/fedora-latest-bazel-coverage-default/log-1e4dde2f-0280-48fb-a945-5fa40b8a3afa.txt
[ RUN      ] ComputeIntegrationTest.CreateDisks
google/cloud/compute/integration_tests/compute_integration_test.cc:126: Failure
Value of: delete_disk
Expected: code is equal to OK and message is anything
  Actual: 256-byte object <70-E8 15-C4 0E-7F 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 ... 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00>, whose status is UNAVAILABLE: Error in non-idempotent operation: Authentication backend unavailable. error_info={reason=backendError, domain=global, metadata={gcloud-cpp.retry.function=DeleteDisk, gcloud-cpp.retry.reason=non-idempotent, gcloud-cpp.retry.original-message=Authentication backend unavailable., http_status_code=503}}, with a code that isn't equal to OK, but a message that is anything

This is a Compute problem, in a Create* request. We could change the test to always retry, ignoring idempotency. Or we could see if something like https://github.com/googleapis/google-cloud-cpp/issues/14255 can be made to work.

coryan avatar Jul 03 '24 14:07 coryan