Logging stops after google.logging.v2.LoggingServiceV2 timeout

[mmmulani]

Environment details

• OS: Ubuntu 20, App Engine
• Node.js version: 20.9.0
• @google-cloud/logging version: 10.5.0
• @grpc/grpc-js version: 1.8.22
• google-gax version: 3.6.1

Steps to reproduce

1. Run server on Google App Engine
2. After some number of hours, we see an error in the logging of:

GoogleError: Total timeout of API google.logging.v2.LoggingServiceV2 exceeded 60000 milliseconds before any response was received.
  at repeat (/app/node_modules/google-gax/src/normalCalls/retries.ts:84:23)
  at Timeout._onTimeout (/app/node_modules/google-gax/src/normalCalls/retries.ts:125:13)
  at listOnTimeout (node:internal/timers:573:17)
  at processTimers (node:internal/timers:514:7) {

From this point onwards, no logs from our app are written and visible in the Log Explorer. Notably, followed the steps from https://cloud.google.com/logging/docs/agent/logging/troubleshooting and it seems like fluentd is still alive, as we are able to see traces of later requests in the Log Explorer but no app logging from them.

This seems like the same issue as 617, though in that case upgrading the package versions helped. I have included all the versions I am using at the top.

Is there some way to catch this error and retry logging overall? Seems odd that the logging service dies completely after one failed request.