google-cloud-rust
google-cloud-rust copied to clipboard
Run the `storage-random` test for one billion requests
This is a smoke test to catch leaks, unexpected race conditions, deadlocks, etc.
With far fewer than 1B requests I got this error:
2025-12-03T01:44:28.914357Z ERROR ThreadId(08) storage_random: error on range 0: the transport reports an error: cannot recover from an underlying read error: the service reports an error with code UNKNOWN described as: error reading a body from connection
Caused by:
0: cannot recover from an underlying read error: the service reports an error with code UNKNOWN described as: error reading a body from connection
1: the service reports an error with code UNKNOWN described as: error reading a body from connection
2: status: 'Unknown error', self: "error reading a body from connection"
3: error reading a body from connection
4: stream error received: unexpected internal error encountered
That sounds like a problem in gRPC that should be recoverable with a retry / resume of some sort.
I am also getting zero bytes transferred, but with more than zero bytes requested. The benchmark does not have enough information to troubleshoot this.
I am also getting:
2025-12-07T23:18:32.785051Z ERROR ThreadId(326) google_cloud_storage::read_resume_policy: unknown error in read resume: Error { kind: Service(ServiceDetails { status_code: None, headers: Some({}), status: Status { code: Unknown, message: "error reading a body from connection", details: [] } }), source: Some(Status { code: Unknown, message: "error reading a body from connection", source: Some(hyper::Error(Body, Error { kind: Reset(StreamId(5), INTERNAL_ERROR, Remote) })) }) }
That seems like a recoverable problem.
The latest run got about 450,000 errors:
grep -c ,ERROR bm-${TS}.txt
442999
``
The only error that was not recoverable is documented in #4069