UnaryCallable.call(request) never returns if access token cannot be obtained

[devchas]

I am working on this bug in the Google Ads client library for Java. I have narrowed the cause to the fact that UnaryCallable.call(request) never returns when instantiated with RequestT of SearchGoogleAdsRequest and ResponseT of GoogleAdsServiceClient.SearchedPageResponse IF the access token cannot be created (e.g. if the refresh token is incorrect).

To reproduce, try running this example with an invalid refresh token.

The issue does not exist with other types of RequestT and ResponseT even with an invalid refresh token (e.g. UnaryCallable<MutateCampaignsRequest, MutateCampaignsResponse>).

This issue may actually stem from the OAuth library used in this library. However, I've tried tracing the issue, and my lack of knowledge regarding this codebase makes it challenging to actually see what's going on once the call method is invoked.

[devchas]

Just checking in on this.

[nwbirnie]

Hey folks, just following up on this. Is it possible to investigate? We have a workaround available which is to use streaming instead of paging RPC. It would also be fine if this is resolved in the Java microgenerator.

[hkdevandla]

@miraleung , any updates on this?

[miraleung]

We're tracking this in the microgenerator and will address it after code completion.

[nwbirnie]

Hey Mira, did you get a chance to look at this one?

[miraleung]

Not yet for the microgenerator, but I can pick this up now. Could you please provide some repro snippets and env setup so I could try this out?

[nwbirnie]

Sure, here's a project that triggers the issue.

Run with ./gradlew run.

I also grabbed a thread dump of it hanging on my machine in case it's helpful.

[nwbirnie]

Friendly ping :)

[miraleung]

Thanks for the ping. We have a Fixit next week, so I've earmarked this for more digging then.

[miraleung]

I tried out the sample with invalid credentials from a different library (which is the same approach used in gax-java's ClientContextTest). In this case, it fails immediately with the error below and does not hang. As such, this is looking like an OAuth bug to me rather than a gax-java issue.

Please feel free to reopen if you think we should dig into this more.

Exception in thread "main" com.google.api.gax.rpc.UnauthenticatedException: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
        at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:73)
        at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72)
        at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60)
        at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
        at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)
        at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1041)
        at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
        at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1215)
        at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)
        at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771)
        at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:563)
        at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:533)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:413)
        at io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:742)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:721)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        Suppressed: com.google.api.gax.rpc.AsyncTaskException: Asynchronous task failed
                at com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:57)
                at com.google.api.gax.rpc.UnaryCallable.call(UnaryCallable.java:112)
                at com.google.ads.googleads.v6.services.GoogleAdsServiceClient.search(GoogleAdsServiceClient.java:170)
                at com.google.ads.googleads.v6.services.GoogleAdsServiceClient.search(GoogleAdsServiceClient.java:159)
                at Issue.main(Issue.java:74)
Caused by: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
        at io.grpc.Status.asRuntimeException(Status.java:533)
        ... 14 more

[nwbirnie]

Hey pretty sure that this is an issue with GAPIC retries. I've added logging to the test project and you can see one error message repeats (I'm sure it will eventually give up). It looks like an issue with retrying failed OAuth requests... we shouldn't be attempting to retry these, the OAuth server is returning 400 bad request which should immediately fail the GAPIC call.

Logs here Updated test project here

[nwbirnie]

More generally I think there's some subset of the OAuth errors that we should never retry. This is one of them (invalid credentials), but there could be others such as invalid grant (a revoked token).

[miraleung]

Thanks for the updates Nick, I'll take another look.

[miraleung]

Sijun will dig into this further - thank you!

[arithmetic1728]

I think I find the root cause. Filed an issue in grpc-java, https://github.com/grpc/grpc-java/issues/8003

[meltsufin]

This is not an issue GAPIC. GAPIC retries whatever codes are "retryable".