Goma HTTP Errors Due to Authorization

[dsanders11]

I've noticed quite a few HTTTP errors when Goma is being used in cache-only mode. When doing a build with the test configuration I'm seeing something like 4-6% of Goma tasks failing with an HTTP error. Seems like they're all cache hits except for these failures.

In the Goma logs, here's an example of the issue, under warnings:

W1015 09:59:23.147698 17864 http.cc:345] Task:566 Throttled backoff=857.041522ms remaining=9m59.999998s
W1015 09:59:24.011807 17864 compile_task.cc:1167] Task:566 rpc err=-3  failed authorization not available
W1015 09:59:24.011807 17864 compile_task.cc:4499] Task:566 no retry: RPC failed http=0 state=PENDING: authorization not available

~~Could this be an issue on the server side?~~

[dsanders11]

Adding more information discussed on Slack. Looks like the failures are related to oauth2 failures, with the goma client trying to refresh authorization when it shouldn't causing the failures:

http status:
 0: 367 (9.242%)
 200: 3913 (98.5394%)
 Retry: 0 (0%)
 Throttled: 341 (8.58726%)
 Connect Failed: 1 (0.0251826%)
 OAuth2 Token Refreshed: 342 (8.61244%)
 Timeout: 0 (0%)
 Error: 367 (9.242%)
 Pending: 341 (8.58726%) peek 14

Seems the main issue is on the client rather than the server, with the client entering a throttled backoff if any HTTP errors occur, and then the authorization error is also considered an HTTP error so it's a positive feedback loop.

A possible fix would be to patch the client to not try authorization any time it's in a throttled backoff.