storage: "before all" hook in "storage" failed

[flaky-bot[bot]]

Note: #1999 was also for this test, but it was closed more than 10 days ago. So, I didn't mark it flaky.

---

commit: 9741a7a2acb2d4b75020d79e34a7d661ab866c63 buildURL: Build Status, Sponge status: failed

Test output

4 DEADLINE_EXCEEDED: Deadline expired before operation could complete.
Error: 4 DEADLINE_EXCEEDED: Deadline expired before operation could complete.
    at Object.callErrorFromStatus (node_modules/@grpc/grpc-js/build/src/call.js:31:19)
        -> /workspace/node_modules/@grpc/grpc-js/src/call.ts:81:17
    at Object.onReceiveStatus (node_modules/@grpc/grpc-js/build/src/client.js:195:52)
        -> /workspace/node_modules/@grpc/grpc-js/src/client.ts:358:36
    at Object.onReceiveStatus (node_modules/@grpc/grpc-js/build/src/client-interceptors.js:365:141)
        -> /workspace/node_modules/@grpc/grpc-js/src/client-interceptors.ts:462:34
    at Object.onReceiveStatus (node_modules/@grpc/grpc-js/build/src/client-interceptors.js:328:181)
        -> /workspace/node_modules/@grpc/grpc-js/src/client-interceptors.ts:424:48
    at /workspace/node_modules/@grpc/grpc-js/build/src/call-stream.js:188:78
        -> /workspace/node_modules/@grpc/grpc-js/src/call-stream.ts:330:24
    at processTicksAndRejections (internal/process/task_queues.js:79:11)
for call at
    at ServiceClientImpl.makeUnaryRequest (node_modules/@grpc/grpc-js/build/src/client.js:163:34)
        -> /workspace/node_modules/@grpc/grpc-js/src/client.ts:328:30
    at ServiceClientImpl. (node_modules/@grpc/grpc-js/build/src/make-client.js:105:19)
        -> /workspace/node_modules/@grpc/grpc-js/src/make-client.ts:189:15
    at /workspace/node_modules/@google-cloud/pubsub/build/src/v1/publisher_client.js:214:29
        -> /workspace/node_modules/@google-cloud/pubsub/src/v1/publisher_client.ts:305:25
    at /workspace/node_modules/google-gax/build/src/normalCalls/timeout.js:44:16
        -> /workspace/node_modules/google-gax/src/normalCalls/timeout.ts:54:13
    at repeat (node_modules/google-gax/build/src/normalCalls/retries.js:80:25)
        -> /workspace/node_modules/google-gax/src/normalCalls/retries.ts:104:19
    at /workspace/node_modules/google-gax/build/src/normalCalls/retries.js:118:13
        -> /workspace/node_modules/google-gax/src/normalCalls/retries.ts:144:7
    at OngoingCall.call (node_modules/google-gax/build/src/call.js:67:27)
        -> /workspace/node_modules/google-gax/src/call.ts:81:23
    at NormalApiCaller.call (node_modules/google-gax/build/src/normalCalls/normalApiCaller.js:34:19)
        -> /workspace/node_modules/google-gax/src/normalCalls/normalApiCaller.ts:43:15
    at /workspace/node_modules/google-gax/build/src/createApiCall.js:84:30
        -> /workspace/node_modules/google-gax/src/createApiCall.ts:118:26
    at runMicrotasks ()
    at processTicksAndRejections (internal/process/task_queues.js:97:5)

[ddelgrosso1]

This seems to be the result of some long standing issues with grpc-js which is utilized by pubsub. See https://github.com/googleapis/nodejs-pubsub/issues/770 as an example. The best path forward is to determine if we really need to be utilizing pubsub in these tests and remove if not. I am going to downgrade this as it seems to only happen on a rare occasion.

[ddelgrosso1]

Spoke with @feywind regarding this. Internal tracking issue #254095632.

[ddelgrosso1]

This is still being investigated internally by the PubSub team.

[ddelgrosso1]

Per the internal tracking ticket this is likely due to network issues and not because of any underlying problem. The recommendation is to roll retries at our level around the failing call. Since this issue happens so infrequently I am going to close this for now. If we see this issue reoccurring we can go ahead and implement retries at that point in time.