Investigate Intermittent GCP Errors

[sklose]

There a few different intermittent errors that have a tendency to pop up when using cc-gpc.

{"log":{"time":"2019-10-04T14:58:42.716Z","level":"EROR","ctx":"XXX","fields":{"msg":["failed to dispatch message"],"type":["SomeEvent"],"err":["Backend Error"],"stack":["Error: Backend Error
    at Util.parseHttpRespBody (/usr/src/node_modules/@google-cloud/common/build/src/util.js:194:38)
    at Util.handleResp (/usr/src/node_modules/@google-cloud/common/build/src/util.js:135:117)
    at retryRequest (/usr/src/node_modules/@google-cloud/common/build/src/util.js:430:22)\\n    at onResponse (/usr/src/node_modules/retry-request/index.js:206:7)
    at /usr/src/node_modules/teeny-request/build/src/index.js:193:17
    at \u003canonymous\u003e
    at process._tickCallback (internal/process/next_tick.js:188:7)"]}},"stream":"stdout","time":"2019-10-04T14:58:42.716629081Z"}

Investigate if it's possible to update the gpc client to handle those errors with a retry mechanism? Are these errors that we can resolve by making changes on our end (e.g. updating node http settings, etc)? Investigate these errors on the gcp client github page to see what could be causing it.

[chandanRIT]

There is also the case when the gcp client just hangs which in turn leaves the whole service hanging. It would be great if we can have a timeout specified somewhere.