google-cloud-rust icon indicating copy to clipboard operation
google-cloud-rust copied to clipboard

Run the `storage-random` test for one billion requests

Open coryan opened this issue 1 month ago • 3 comments

This is a smoke test to catch leaks, unexpected race conditions, deadlocks, etc.

coryan avatar Dec 02 '25 14:12 coryan

With far fewer than 1B requests I got this error:

2025-12-03T01:44:28.914357Z ERROR ThreadId(08) storage_random: error on range 0: the transport reports an error: cannot recover from an underlying read error: the service reports an error with code UNKNOWN described as: error reading a body from connection

Caused by:
    0: cannot recover from an underlying read error: the service reports an error with code UNKNOWN described as: error reading a body from connection
    1: the service reports an error with code UNKNOWN described as: error reading a body from connection
    2: status: 'Unknown error', self: "error reading a body from connection"
    3: error reading a body from connection
    4: stream error received: unexpected internal error encountered

That sounds like a problem in gRPC that should be recoverable with a retry / resume of some sort.

coryan avatar Dec 03 '25 14:12 coryan

I am also getting zero bytes transferred, but with more than zero bytes requested. The benchmark does not have enough information to troubleshoot this.

coryan avatar Dec 05 '25 14:12 coryan

I am also getting:

2025-12-07T23:18:32.785051Z ERROR ThreadId(326) google_cloud_storage::read_resume_policy: unknown error in read resume: Error { kind: Service(ServiceDetails { status_code: None, headers: Some({}), status: Status { code: Unknown, message: "error reading a body from connection", details: [] } }), source: Some(Status { code: Unknown, message: "error reading a body from connection", source: Some(hyper::Error(Body, Error { kind: Reset(StreamId(5), INTERNAL_ERROR, Remote) })) }) }

That seems like a recoverable problem.

coryan avatar Dec 08 '25 15:12 coryan

The latest run got about 450,000 errors:

grep -c ,ERROR bm-${TS}.txt
442999
``

The only error that was not recoverable is documented in #4069 

coryan avatar Dec 14 '25 17:12 coryan