google-cloud-rust
google-cloud-rust copied to clipboard
googleapis
Flake in Storage integration test
Created too many buckets, probably without the right retry policy:
2025-10-17T15:10:26.737575Z ERROR ThreadId(126) integration_tests: ERROR the service reports an error with code RESOURCE_EXHAUSTED described as: Create Bucket quota limit limit exceeded on the Google Cloud project. Please slow down execution and try again.
--
|
| Caused by:
| the service reports an error with code RESOURCE_EXHAUSTED described as: Create Bucket quota limit limit exceeded on the Google Cloud project. Please slow down execution and try again.
| test driver::run_storage_upload_buffered::default ... FAILED
https://pantheon.corp.google.com/cloud-build/builds;region=us-central1/3ee53345-ada3-44fa-b9c1-f901ff3d2a58?project=rust-sdk-testing&e=SqlStudioMysqlLaunch::SqlStudioMysqlEnabled&mods=monitoring_api_staging
We seem to have mitigated this problem. Let's wait 90d and close this issue if there are no more problem
Still flaking:
---- storage::driver::run stdout ----
--
| cleaning up 0 buckets
| object successfully uploaded Object { name: "test-only", bucket: "projects/_/buckets/rust-sdk-testing-jdcblq8josgd1j1mpliopa1v8k4gkxik6tyddeafsvcwf5", etag: "CMzE7eaIvZADEAE=", generation: 1761316911669836, restore_token: None, metageneration: 1, storage_class: "STANDARD", size: 41000, content_encoding: "", content_disposition: "", cache_control: "", acl: [], content_language: "", delete_time: None, finalize_time: Some(Timestamp { seconds: 1761316911, nanos: 683000000 }), content_type: "", create_time: Some(Timestamp { seconds: 1761316911, nanos: 683000000 }), component_count: 0, checksums: Some(ObjectChecksums { crc32c: Some(882148875), md5_hash: b"K\x91rz)-\xaf'\xeb\x85\xf3\xaf*\x91\x99[" }), update_time: Some(Timestamp { seconds: 1761316911, nanos: 683000000 }), kms_key: "", update_storage_class_time: Some(Timestamp { seconds: 1761316911, nanos: 683000000 }), temporary_hold: false, retention_expire_time: None, metadata: {}, contexts: None, event_based_hold: None, owner: None, customer_encryption: None, custom_time: None, soft_delete_time: None, hard_delete_time: None, retention: None }
| PROGRESS: total_bytes_rewritten=1048576; object_size=3145728
| PROGRESS: total_bytes_rewritten=2097152; object_size=3145728
| DONE: total_bytes_rewritten=3145728; object_size=3145728
| dest_object=Object { name: "rewrite-object-clone", bucket: "projects/_/buckets/rust-sdk-testing-jdcblq8josgd1j1mpliopa1v8k4gkxik6tyddeafsvcwf5", etag: "CPHNmOeIvZADEAE=", generation: 1761316912375537, restore_token: None, metageneration: 1, storage_class: "NEARLINE", size: 3145728, content_encoding: "", content_disposition: "", cache_control: "", acl: [], content_language: "", delete_time: None, finalize_time: Some(Timestamp { seconds: 1761316912, nanos: 380943000 }), content_type: "", create_time: Some(Timestamp { seconds: 1761316912, nanos: 380000000 }), component_count: 0, checksums: Some(ObjectChecksums { crc32c: Some(1755103855), md5_hash: b"\xdc\xe39\xbc\xcb\xe038\x08k\xb3C 4\xa3E" }), update_time: Some(Timestamp { seconds: 1761316912, nanos: 380000000 }), kms_key: "", update_storage_class_time: Some(Timestamp { seconds: 1761316912, nanos: 380000000 }), temporary_hold: false, retention_expire_time: None, metadata: {}, contexts: None, event_based_hold: Some(false), owner: None, customer_encryption: None, custom_time: None, soft_delete_time: None, hard_delete_time: None, retention: None }
| dest_object=Object { name: "rewrite-object-clone", bucket: "projects/_/buckets/rust-sdk-testing-jdcblq8josgd1j1mpliopa1v8k4gkxik6tyddeafsvcwf5", etag: "CPamvOeIvZADEAE=", generation: 1761316912960374, restore_token: None, metageneration: 1, storage_class: "STANDARD", size: 3145728, content_encoding: "", content_disposition: "", cache_control: "", acl: [], content_language: "", delete_time: None, finalize_time: Some(Timestamp { seconds: 1761316912, nanos: 964767000 }), content_type: "", create_time: Some(Timestamp { seconds: 1761316912, nanos: 964000000 }), component_count: 0, checksums: Some(ObjectChecksums { crc32c: Some(1755103855), md5_hash: b"\xdc\xe39\xbc\xcb\xe038\x08k\xb3C 4\xa3E" }), update_time: Some(Timestamp { seconds: 1761316912, nanos: 964000000 }), kms_key: "", update_storage_class_time: Some(Timestamp { seconds: 1761316912, nanos: 964000000 }), temporary_hold: false, retention_expire_time: None, metadata: {}, contexts: None, event_based_hold: Some(false), owner: None, customer_encryption: None, custom_time: None, soft_delete_time: None, hard_delete_time: None, retention: None }
| Uploaded object 1MiB.txt, size=1024KiB
| Created object 1MiB.txt, size=1MiB
| Created object 1MiB.txt, size=1MiB
| Created object 2GiB.txt size=2048 MiB
| Error: the service reports an error with code INTERNAL described as: We encountered an internal error. Please try again.
May need to improve the logging too.